Predicting sensory performance [Elektronische Ressource] : maps, optimality, feature extraction, and learning / Andreas B. Sichert

131 Pages

English

Free trial

Gain access to the library to view online
Learn more

Predicting sensory performance [Elektronische Ressource] : maps, optimality, feature extraction, and learning / Andreas B. Sichert

technische_universitat_munchen

Free trial

Gain access to the library to view online
Learn more

131 Pages

English

About
Information
Extrait

Description

Predicting Sensory PerformanceMaps, Optimality, Feature Extraction, and LearningAndreas B. SichertcCover illustration: Photograph courtesyof Guido Westho Autor: Andreas B. SichertTitle: Predicting Sensory PerformanceSubtitle: Maps, Optimality, Feature Extraction, andLearningFunding has been provided by the Bernstein Centerfor Computational Neuroscience (BCCN) { Munich,the CoTeSys Exzellenz Cluster, and the Sloan-SwartzFoundation.Physik DepartmentTechnische Universität MünchenPredicting Sensory PerformanceMaps, Optimality, Feature Extraction, and LearningAndreas B. SichertVollst andiger Abdruck der von der Fakult at fur Physik der Technischen Universit at Munc henzur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften genehmigtenDissertation.Vorsitzender: Univ.-Prof. Dr. Th. HugelPrufer der Dissertation: 1. Dr. J.L. van Hemmen2. Dr. R. NetzDie Dissertation wurde am 14.07.2010 bei der Technischen Universit at Munc hen eingereicht unddurch die Fakult at fur Physik am 20.09.2010 angenommen.Prefaceow, you are reading my PhD thesis! I mean, you are reading { which is an almost unbelievablyWcomplex process. It is, however, a good example of what makes neuroscience so fascinating.Let’s take a closer look at the above example of reading to understand what neuroscience is allabout.At rst glance, reading starts with some tiny black dots on a white sheet of paper which formgeometrical structures that we denote as letters.

Subjects

Documents

Knowledge

Physik

Informations

Published by	technische_universitat_munchen
Published	01 January 2010
Reads	13
Language	English
Document size	23 MB

Exrait

Predicting

Maps,

rySenso

yitOptimal,

eatureF

Andreas

rmanceerfoP

Extraction,

Sichert

nda

rningLea

Photograph

c

courtesy

Coverillustration:Photographccourtesy
esthoﬀWGuidoofTitle:Autor:PredictingAndreasB.SicSensoryhertPerformance
Subtitle:Maps,Optimality,FeatureExtraction,
Learning

and

FundinghasbeenprovidedbytheBernsteinCenter
theforCoTeSysComputationalExzellenzNeuroscCluster,ienceand(BCCN)the–Sloan-SwMunich,artz
ion.taoundF

PredictingSensoryPerformance
Maps,Optimality,FeatureExtraction,andLearning

B.AndreasSichert

Vollst¨andigerAbdruckdervonderFakult¨atf¨urPhysikderTechnischenUniversit¨atM¨unchen
zurErlangungdesakademischenGradeseinesDoktorsderNaturwissenschaftengenehmigten
Dissertation.

Vorsitzender:Univ.-Prof.Dr.Th.Hugel
Pr¨uferderDissertation:1.Univ.-Prof.Dr.J.L.vanHemmen
NetzR.Dr.Univ.-Prof.2.

DieDissertationwurdeam14.07.2010beiderTechnischenUniversit¨atM¨uncheneingereichtund
durchdieFakult¨atf¨urPhysikam20.09.2010angenommen.

Preface

Wow,complexyouareprorecess.adingItmis,yhoPhDwevtheser,ais!goIodmean,eyxampleouareofwhatreading–makeswhichneurisosanciencealmostsounbelievfascinating.ably
Let’stakeacloserlookattheaboveexampleofreadingtounderstandwhatneuroscienceisall
out.abAtﬁrstglance,readingstartswithsometinyblackdotsonawhitesheetofpaperwhichform
geometricalstructuresthatwedenoteasletters.Wecanperceivethembecauseoureyesandthe
subsequentneuronalprocessingofthevisualsystemareabletotranslatepicturesintheexternal
worldintoneuronalinformationthatformsacorrespondingrepresentationwithinourbrain.The
visualsystem,however,doesnotsettleinformationprocessingbypresentingcloudsofblackdots.
Atgroupthisthepoint,ineuronalnformationrepreseprontationcessingofstartsthesetocloudsgetreallyofdotstosophisticated.mentalobjeCognitivcts–enetheuronalletters.circuitsThe
activitiesindividualetc.lettersToareﬁnallyrelategaindatoeacreasonablehothermetoaningform,wtheordsbrainthatusestherepresenundterreallyingobjectsgrammar,attributes,thatis
thespeciﬁcmethodologyoftheparticularlanguagetobuildsentencesoutofindividualwords.
andForsenanytencesonearewhoisonlynotalreadymeaningfulbimpressedecauseletwemeprelateointtheoutsetwocloudsfurtherofdotsissues.andFirst,thecorrespletters,wondingords,
abstractmentalobjectstopastsensoryexperience.Wecaninterprettheword“barking”because
oursomevisualtimesysago.temThisrelatesisduethetoblacthekfactdotsonthatthediﬀsheeterentofmopapdalitieseratarehandabtoletotheinsoundteractofafaithfudoglly,heardfor
instance,bymeansofcombiningthecorrectmemoriesofvisualandauditorysensoryevents.
Second,theanswertothequestionofwheresuchintelligentneuronalcircuitscomefromis
elusive.Mayevolutionbeananswer,viz.,aretheseneuronalstructuresencodedinourgenes?
No,notintheﬁrstplace.OtherwisetherewouldbegenesforBavarian,German,English,and
otherlanguages.Theabilitytoread,tointerpretabstractmethodologyandsymbols,islearned.
Moreover,thereisnoteacherthatgivesbabieslecturesintheirmothertongue.So,ourbrain
teachesitself,thatisitlearnspurelyfromsensoryexperiencetobuilditselfinawaytobebest
suitedforComparingprocessingthesesensoryabilities–expourerience.abilities–withthoseofman-madeandsocalled“cognitive”
Fromsystemsmyispersonalimpressivpe.erspInectivconethtrastatistowhnatureatmakweeshatheve,ﬁeldhowofever,justneurosciencestartedsotoinfascinating:ventsuchDiscovsystems.ering
thenervoussystem,ahellofamachinery!

ofdiﬀTheerentotolsfordisciplinesdiscovisneringethecessarynervtoousenlighsystemtenthisarecomplexmanifold.topic.ItturnsByoutmeansthatofathisbroadthesisintewreaclotionok
atsituatedthenervatoustheinsystemtersectionthroughofthebiology,glassesofmedicine,theoreticandalphysicsneuroscience.employingTheoreticalmathematicalneurosciencetools.Inis
mashort,jorthisdrivingdissertationforceforfothecuseswholeon“sPystemredicting–includingSensoryPitsdeverformance”elopment.sinceForsensorythatpuexprperieose,nceweiswillthe
themderivetospeciﬁmathematicalcsettings,viz.,descriptionsdiﬀoferentsensorysenses.systems,featureextractionandlearning,andapply

Preface

1ChapterTheﬁrstchaptergivesanintroductiontofundamentalconceptsofsensoryprocessingsuchasthat
ofaneuronalmapwhichareneededtounderstandtherestofthework.Besides,thereferences
giveninthecompactintroductionshouldprovidethereaderwithagoodstartingpointforown
studies.

2ChapterInordertoexploittheinformationprovidedbysensorysystems,animalsreconstructthespatio-
temporalenvironmentfromtheirsensorysystems.Inthischapterwederiveauniversalframework
thatallowstocalculatethespeciﬁclayoutoftheinvolvedneuronalnetworkbymeansofa
generaltransmissionmathematicalandrudimenprinciple,talknoviz.,wledgestocofhasticthedetectionoptimality.procThatess,is,thisgivenapproacaknohwnallophwstoysicalestimatesignal
thepossibleperformanceandtopredictneuronalpropertiesofbiologicalsensorysystems.

3ChapterInthischapterweapplyourframeworkderivedinChap.2totheauditorysystem.Thatis,
comingfromthegeneralmathematicalprincipleofoptimalitywededuceanarchetypemodeland
correspondingtemporalreceptiveﬁeldsforauditorysignalenhancementandechosuppression.We
comparetheperformanceofboththeneuronalandthemathematicalimplementationofthemodel
tostate-of-the-artalgorithmsandbiologicalsolutions.

4ChapterThelateral-linesystemisauniquemechano-sensoryfacilityofaquaticanimalsthatenablesthem
notonlytolocalizeprey,predators,obstacles,andconspeciﬁcsbutalsotorecognizehydrodynamic
objectsaswedemonstrateinthischapter.Wepresentbothananalyticapproachthatshowsthe
eﬀectofsubmergedmovingobjectshapeandanexplicitmodelexplaininghowaquaticanimalscan
distinguishdiﬀerentshapes.Furthermore,wepredictnaturallimitationsofthissensorycapability.

5ChapterTwogroupsofsnakespossessaninfrareddetectionsystemtoperceiveaheat-imageoftheir
environment.Thischapterexploresthediﬀerentstagesoftheinfraredpathwaysuchastransmission,
detection,andneuronalen-anddecodinginordertoprovidefundamentalinsightstothissensory
system.Thethoroughphysicaldescriptionoftheinvolvedphenomenaaswellastheincorporation
ofanatomicaldataintoanexplicitmodelextendthepresentpictureofthissensorysystem.

6ChapterForneuronalprocessingofsensoryinformationuni-andmultimodalmapsplayakeyrole.The
preciseinteractionofsuchmapsisnotpresentatbirthbuthastobelearnedduringthedevelopment
ofananimalbymeansofsensoryexperience.Thischapterpresentstheconceptofintegrated
MultimodalTeachingthatcombinessupervisedspike-timing-dependentplasticityandmultimodal
integrationtoinherentlydevelopafaithfulinteractionofdiﬀerentsensorymodalities.

yThankou

Preface

ManypeoplehavesupportedmeduringmytimeatthePhysikDepartmentoftheTechnische
Universit¨atM¨unchen.Intheﬁrstplace,IthankLeovanHemmen,mythesissupervisor.Hehas
introducedmetothetopicoftheoreticalneuroscience,viz.,biophysics.Hisdedicationtoscientiﬁc
researchisveryimpressive,andthroughmanyinspiringdiscussionsIhavelearnedalotabout
science.

Writingthisthesiswouldnothavebeenpossiblewithoutthepleasantinteractionandcooperation
mawithysamyythankcolleaguesyoutoattheMoritzB“T35”¨urcck,hairStefanieandaB¨urcfewk,pReopleolandbeyUtz,ond.InAndreasthecVonollmatextyr,ofandthisthesis,ChristineI
Voßenwhohaveproofreadpartsofthethesisandhavegivenhelpfulcomments.

Asawhole,Iwasgiventheopportunitytomakemanyvaluableexperiencesduringthetimeof
mUnivyPhDersity.wIork;havinealsoaddition,meteIwxceptionasablealptoeoplejoinandthehaBavyeeriscmadehenewEliteAkfriends.ademieForandalloftothisvisitIamHarvvardery
grateful.

Finally,Ithankmyfriends,myparents,mybrother,andBabsi,whohavealwaysbeenthereto
me.ortsupp

2010Juneh,Munic

vii

Contents

PrefacetstenConInformationSensorycessingPro11.1Stepsofsensoryprocessing...............................
1.1.1Signalmapping..................................
1.1.2Sensoryencoding.................................
1.1.3Neuronaldecoding................................
1.2Neuronalmaps......................................
1.2.1Mapproperties..................................
1.2.2Collectionofcomputationalmaps........................
1.2.3Mapdevelopment................................
1.3Learninginsensorysystems...............................
1.3.1Localmechanisms................................
1.3.2Globalstructures.................................
ormationFMapOptimal22.1Startingpoint.......................................
2.2Mathematicalmodel...................................
2.2.1Deﬁnitionoftheproblem............................
2.2.2Optimalreconstruction.............................
2.2.3Matrixnotation.................................
2.3Relationtocommonmethods..............................
2.3.1Pseudo-inverse...................................
2.3.2Maximum-likelihoodapproach..........................
2.4Neuronalrealizationofthemodel............................
2.5Multimodalinteraction..................................
2.5.1Integration....................................
2.5.2Pooling......................................
2.5.3Development...................................
3Auditoryprocessing:Exemplifyingoptimality
3.1Introduction........................................
3.2Archetypemodel.....................................
3.2.1Optimality:Anarchitecturalprinciple.....................
3.2.2Receptiveﬁelds:Acomputationalbuildingblock...............
3.3Technicalprospect....................................
3.4Biologicalprospect....................................
3.5Discussion.........................................

vix112234568991113131314151818181920232323242525262627293132

Contents

35erceptionPLateral-line44.1Introduction........................................35
4.2Analyticdescriptionofshape..............................36
4.2.1Mathematicalformulation............................37
4.2.2Eﬀectsofshape..................................37
4.3Hydrodynamicobjectrecognition............................40
4.3.1Deﬁnitionofhydrodynamicobjects.......................40
4.3.2Objectrecognitionperformance.........................42
4.4Discussionandoutlook..................................46
5InfraredSystemofSnakes49
5.1Introduction........................................49
5.2Warmingup:Signalmapping..............................51
5.2.1Athermodynamicvista.............................51
5.2.2Mappingtheheat................................57
5.3Sensoryencoding.....................................59
5.4Neuronaldecoding....................................63
5.4.1Minimalmodel..................................63
5.4.2Theextendedmodel...............................67
5.4.3Featureextraction................................69
5.5DiscussionandOutlook.................................73
6IntegratedMultimodalTeaching75
6.1Introduction........................................75
6.2IntegratedMultimodalTeaching(iMT).........................76
6.2.1HowdounisensorymapsdetermineiMT?...................77
6.2.2HowdoiMTcharacteristicsinﬂuencemapadaptation?............79
6.2.3HowdoesiMTcalibratediﬀerentunimodalmaps?..............84
6.3Modelresultsandexperimentalﬁndings........................84
6.3.1Experimentsprovision-guidedmapformation.................85
6.3.2Experimentscontravision-guidedmapformation...............87
6.4DiscussionandOutlook.................................87
AAppendix:OptimalMapformation95
A.1Arecipeformakingmaps................................95
A.2Nonlinearitiesininformationprocessing........................95
A.3Remainingderivationstepsleadingto(2.16)......................96
A.4Gaussianblurredsignal.................................98
A.5Pseudoinverse.......................................99
BAppendix:Lateral-linePerception101
B.1Planepotentialﬂow...................................101
B.2Rotational-symmetricpotentialﬂow..........................101
103iMTendix:AppCC.1Optimalcombinationoftwomodalities.........................103
C.2Optimalcombinationofthreemodalities........................104
C.3ModelParameters....................................105

bDieeginntReisezuvondeinentausendF¨ußen.Meilen

1.ProcessingSensoryInformation

Laotse

Thisthroughchapterthepglassesrovidesofthisawosynopsisrk,weofapplythewideOccam’sarearazoofrbysensorymeansinfoofintrormationducingproandcessing.foLocussingokingon
inthethesessentialubsequentconceptschapters.usedinItthisstartsthesiswithenabthelingusquestiontoofderivewhatparedictionsstimulusoninsensoryexternalperfoworldrmanceis,
discussestheconceptualstepsinthesensorypathway,andpresentsabriefoverviewofneuronal
repsynapticresentationplasticitbyy,ameanstoolofformaps.understaFinallynding,wefocusdevelopmourentofattentionaoncommonmapfomulti-sensormationryandspaceintrothatduce
enablesbeingstoactfaithfullyintheexternalworld.

1.1Stepsofsensoryprocessing
SensoryprocessingcanbedividedintothreeconceptualstepsasdepictedinFig.1.1(A−C).First,
astimstimulusulusinisaphspeciﬁcysicallywaymappandedontranslatetodetectorthephysicalorgans.signalSecond,intotheneuronaldetectorinforgansormation,respviz.,ondtospikthees
ofseethee.g.corr[164esp,267onding],andprimaryforthesensorydiﬀerentneurons;levelsforofinmotrodelingductionstoneuroscienceneurons,e.g.neuronal[119].Third,computationthe
provideinformationausefulcontainedrepresenwithintationtheofspikthee-trainsoriginalofstimtheulussensoryto,forsysteminstance,isproinitiatecessedmotorneuronallycommands.soasto

Figure1.1:Threeconceptualsteps(A−C)ofsensoryprocessing.Everystimulusintheexternal
worldemitssignals,inourexamplelight.(A)First,thesignalsaremappedontothesensoryorgans
wheretheyaﬀectthephysicalstateofspecialized(protein)structureswithinthe(photo)receptors.
(B)Second,thesensoryneuronsrespondonlytospeciﬁcstimuli,forinstancebluelight,bymeans
ofgeneratingcorrespondingspikes.(C)Third,theneuronalsystemdecodestheimbeddeddata
withinthespike-trainstorevealtheessentialinformationinausefulway.

1.ProcessingSensoryInformation

mappingSignal1.1.1Initially,anyobjectintheexternalworldrevealsitspresencebygeneratingdiﬀerentsignals.A
runninganimalmay,forinstance,generateasoundandachangingvisualimageasitmoves,as
wellasvibrationsandaninfraredproﬁle.Thegenerationandtransmissionprocessfollowsspeciﬁc
physicallaws.Thisfact,assimpleasitmaylookatﬁrstglance,providesapowerfultoolfor
understandingsensoryprocessing:themathematicaldescriptionofsensorystimuliasphysical
tities.quanableobservInconcreteterms,givenanysignalthatvariesxasafunctionofspatialpositionxandtimet,it
ispossibletocalculatethetime-dependentinputi(t)tothereceptororgans.Thatis,thephysical
mappingofsignalssx(t)ontosensorydetectorscanbedescribedbyasetoftransferfunctionsh,
h:sx(t)→ix(t).(1.1)
Moreover,realobjectsintheexternalworldtypicallyemitabundleofsignalsthroughseveral
channelsbasedondiﬀerentphysicalpathwaysatthesametime.Thatis,theypermanentlyleavea
multi-sensoryﬁngerprintthatcanbedetectedandprocessedbysensorysystemstoformacoherent
representationoftheexternalworldinspace-time.Thisissuewillleadustomultimodalintegration
andlaterontothequestion,howthispreciseinterplayofoursensesevolves.Weleavetheissue
ofmultimodalprocessingforthetimebeingandnowresumediscussingthefundamentalstepsof
sensoryprocessingwithinonemodality.

dingencorySenso1.1.2Inthesecondstepofsensoryprocessing,thesignalthatismappedontothesensorydetectors
istranslatedintoneuronalinformation–spikes.Thisprocesscanalsobedescribedbyasetof
oftransferthefudetector.nctionsk.ComingInthebacbktoeginningtheofexampletheproincess,Fig.the1.1,externallightinsignalthevisualaﬀectstherangeph–ysicalsimpliﬁedstate
–changesthestateofthewavelength-sensitivephotoreceptorsintheretina.Bymeansofthis
physicalprocess,spikegenerationisinduced.Theresultingspikesti(t)consequentlyencodethe
intensityofthecorrespondingspeciﬁcwavelength[263,267,281]ataparticulardetectorpositioni
,ttimeandk:ix(t)→ti(t).(1.2)
issueTwisothataspectssensoryofsensoryencodingencodingincludesareatofleastfurthertwoinstepsterestofinsignthealconﬁltetextring.ofthisThephthesis.ysicalThechangeﬁrst
ofstatewithinthereceptorisspeciﬁctoaparticularstimuluscharacteristicandhenceﬁltersthe
signal.Inourexample,thephotoreceptorrespondsonlytobluelight;forreviewofthevisualsignal
cascadeInseeaddition,[263].theSo,prothecesswavofspikelength-speecgenerationiﬁcityalsselectsoﬁltersoutthelightofsignal.allWotherenowcolors,takecf.aclFig.oser1.1.look
atthisimportantissue.First,translatingsensorysignalsintoneuronalinformationcomparesto
ananalog-digital-conversionprocessbecauseaspikecodeisabinarycodeconstitutedofzeros
(nospikesatthispointintime)andones(spikeatthisparticulartime).Second,intheprocess
oflimitspiktheesamplinggenerationrate,atthee.g.levelofrefractiononetimesinglelimitsneuron,thespikbiopheysicalfrequencydyn[am267ics,of164].cellThisrescompartmenultsintsa
restrainedinformationconsamplingtent.Third,accuracy,thethatneuronalis,thetransmissionanalog-digital-conprocessvalsoersionalteprorscesstheimbprincipallyeddedreduceinformation.sthe
Synapses,forinstance,typicallyfeaturecomplexnon-lineardynamicsthatcontributetoaniterative
sequenceoffragmentation,selection,andampliﬁcationofinformation,andthusﬁlterthesignal
[89].information.Fourth,thereNeuronsarecandiﬀerenresptondkindstoofspikeneuronalratesco(“ratedingcothatding”)butadditionallyinothermaycasesmodifyneuronstheresporiginalond

1.ProcessingSensoryInformation

toﬁringonset,changeofspikerates,skipping,bursting,orcomplextemporalpattern–justtolist
afew[18,37,247].Wesee,generatingandtransmittingneuronalinformationpresentsdeﬁnitely
aﬁlteringprocess.Thereisnodoubt,however,thattheneuronalﬁlteringprocessesneednot
generallybebugs.Theymay,onthecontrary,benecessaryfeaturesreducingthearrivingphysical
signalstotheessentialonesandthusmakingexternalsensoryinformationstreamscomputationally
feasible.Thesecondimportantaspectofsensoryencodinginthisthesisistopography.Somesensory
organsareorganizedinsuchawaythattheyareabletoreceiveandconservetopologicallyordered
information.Inourexample,theretinashowsatopologicalorganizationofthedetectorsand
lensesassuringawelldeﬁnedtopologicalrelationbetweenpositionsintheexternalworldxand
thedetectorpositionsi.Translatingspatialinformationfromtheexternalworldintoneuronal
information,however,inducesatransferfromcontinuoustopologiestodiscretetopologies.This
spatialanalog-digital-conversionprocessatleastcausesuncertaintiesinthesensorypathway.
Topography,moreover,isnotonlyaspatialcharacteristic.Itisalsoimportantinthetemporal
domain.Withintheauditorysystemitiscrucialtoknowthetemporalsequenceofeventsinorder
to,forinstance,processinterauraltimediﬀerences.Withinthelimitationofaneuronalsystemour
brainisevenabletotranslateneuronallytemporaldataintospatialinformationtogainauseful
representationofexternalspace.TheJeﬀress’modelisafamousexampleofsuchatranslation
[132].cesspro

dingdecoNeuronal1.1.3InTheythemanextycomestagefofromsensorydiﬀerentprosenscessing,orytheneuronsineuronalwhichmasystemyhasadditionallytodecohadeveathespspikatialore-trainstempti(oralt).
relation.Inmathematicalterms,theneuronalsystemprincipallyhastosolvetheneuronalinverse
andproblem.signalThatmapping)is,theandsystemkhas(sensorytoinvertdetectingthecomandpositionneuronalofencotransferding)tofunctionsrevealhthe(signaloriginalgeneratingsignal
s.Ingeneral,itishardlypossibletoﬁndanexactsolution.Thesystem,however,cantrytoﬁnd
anoptimalinversefunctionltoreconstructsˆtheoriginalsignalfromtheneuronaldata
l:ti(t)→ˆsx(t).(1.3)
Additionally,inmanycasesthesensorysystemevensettlesforcomputingtheessentialstimulus
informationinsteadofreconstructingthecompletestimulus.Energycostsandthelimitationsof
neuronalprocessingcapabilitiesmaybereasonsforthat.
Thelimitationseachsensorysystemisfacedwithleadtoanotherglobalissueinsensory
processing.Asensorysystemcanonlyposeandanswerthefollowingquestion:Givenuncertainties,
sensorylimitations,respandonse?aForsensoryillustration,response–thewhatﬁniteisntheumbermostofprdeteobablectorsissignalnotthaablettocouldsamplehaveconcausedtinuousthis
signalsorspacewithoutuncertainties.Thelimitingtemporalaccuracyofneuronalcomputationas
wellmatterasthewhatneuronalcausesthesenoiselevelsystemadditionallyrestrictions,increasetheproquestioncessingitselfreduceimpreciseness.snaturallyConsequentoaqtlyues,andtionnoof
conditionalprobability.So,itisnotsurprisingthateverywhereinthecourseofsensoryprocessingwe
ﬁndneuronalstructuresbehavinglikeestimators[7,59,61,67,69,113,120,183,220,237,238,239]
that,forinstance,canoptimallysolvetheneuronalinverseproblem[75,76,252].Itremains,
however,torevealhowtheseestimatorsworkonthelevelofneuronalcircuitryandtoquantify
erformance.ptheir

1.ProcessingSensoryInformation

Figure1.2:Threecategoriesofmapsbasedontheirtopography;adaptedfrom[11,154,208].
NelsonandBower[208]suggestthreemapgroups,i.e.continuousmaps(A),patchymaps(B),
andscatteredmaps(C).(A)Mapcorrespondstoacontinuousmappingofauditoryspaceinthe
optictectumofthebarnowlshowingaglobaltopography[154].(B)Thepatchymappingofvisual,
auditory,andmultisensorysensitiveareasinthehumansuperiortemporalsculusmultisensoryarea
showsatopographyonlywithinsmallanddistinctareasofthewholeorganization[11].(C)Inspite
ofthespatialorganizationinthelattertwoexamples,thescatteredmappingofolfactoryinputto
thepiriformcortexoftheratlacksanyspatialorganization.Dependingonthecomputationaltask,
thediﬀerentstructuresleadtodiﬀerentcomputationaladvantagesasbrieﬂyaddressedinthemain
text.

aimInofallsummarythe,theresultingphysicsneuronalofexternaleﬀortissignaltoachievtransdueactionreasonabledictateandtheusefulsensoryrepresedatantationstructure.ofThethe
originalstimulusinformation,viz.,thesignal.Thecentralnervoussystem(CNS)canrepresentthe
thecomputedtopologicallyinformationorderedindiﬀerenneuronaltwamapys.Inplaythisanthesisessenthetialprole.opulationThecomapdeand,accuracymoredeimpﬁnedortanastltoy,
howpreciseamapcanrepresentanexternalstimulusisagoodmeasurefortheperformanceofthe
correspondingsensorysystem.Sowewillconsidermapsinthenextparagraphinmoredetail.

mapsNeuronal1.2Amajorroleinsensoryprocessingisreservedforneuronalmaps1[116,161].Theconceptofamap
suggeststhatthereisalawfulrelationbetweenthesurfaceoftheneuronalsubstrateandarelevant
aspectoftheexternalspace.Tocutalongstoryshort,aneuronalmapistypicallyanarrayof
neuronsinwhichneighboringneuronsrespondto“similar”sensorystimuli.Thissimpliﬁcation
holdstrueatleastforspatialsensorymapsonwhichwefocusinthecourseofthisthesis.
Asanexample,visualinputinthemammalianbrainisprocessedthroughmultiplecorticallayers
thatareorganizedaccordingtothetopographyoftheretinalinputcells(“retinotopicorganization”)
[281,143].Inthisway,neighboringneuronsrespondtovisualinputfromneighboringpointsin
space,andthusformaspatialmap.Spatialmapshavebeendiscoveredinvarioussensorysystems
inmanygroupsofvertebrates[10,40,62,109,123,149,151,192,259,264,295]andeveninsome
[130].ertebratesnon-v1InReihenfolge1879,kHelmholtzennen[gelern114]tremarkwird,inedder“DasssichdurcihrehdasEindr¨Enucketlangf¨darbieten,uhrendesdassdiesetastendenReihenfolgeFingersansichdenalsObunabh¨jectenangigdie
davderenonEerwlemeneist,tenobmanmanmitimmerdiesemwiederodervor-jenemoderrFinger¨uckw¨artstastet,indassderselsiebenfernerOrdnnichungteinedurceinl¨hlaufenauﬁgmbuss,estimmumtevonReiheeinemist,
zumRiemann’sanderenTermizunoklogieommen,,einealsokeineMannigfaltigklinienf¨eitzwormigeeiterReihe,Ordnung,sonderndasalleseinﬂ¨istacleichenhahtfteseinzuseheNebn.”eneinander,Thatis,oderHelmholtznach
alreadyrecognizedthataneuronalrepresentationofatwo-dimensionalsurfaceconstitutesatwo-dimensional
brain.theinmanifold

1.ProcessingSensoryInformation

Figure1.3:Theﬁringproﬁleofaspatialmapencodesthelikelihoodofﬁndinganobjectatacertain
positioninsensoryspace.Focusingontheﬁringrateofonlyasingleneuronsuchasneuroni
withfaithfulppreferrederceptiondirectioncanonlyxibandeachievignoringedifitstheneighactivitbyorsofprevtheentswholeafaithfulneuronalpmaperception(justofcomparereality.theA
“neighborhood”ofiwiththerest)istakenintoaccount.Consequently,eventhoughthenominal
valueoftheﬁringrateofneuroniisidenticalincaseAandB,therepresentedphysicalreality
diﬀerssigniﬁcantlyinbothcases.

Neuronalmapsarenotlimitedtospatialrepresentations.Prominentexamplesarefoundin
theauditorysystem.Therearemapsrepresentingfrequency,interauraltimediﬀerence,interaural
amplitudediﬀerence,andevenamplitudemodulationfrequencies[84,187,214,225,245,265].So,
weseethatmapsareneitherrestrictedtoparticularsensorymodalitiesnorparticulardomains
space.ashsucAtypicalwayofcategorizinginphysicsistosearchforcharacterizinglengthscales.Lookingat
maps,wecanﬁndtwotypesoflengthscales.Firstandobviouslywecanconsiderthedimensions
oftheneuronalarray.FollowingthepresentationofDehaeneandCohen[58],onemayhence
distinguishmapsbytheirsizeintomacromaps(afewcentimeters),mesomaps(uptocentimeters),
andmicromaps(millimeters).Second,besidethelengthscaleofmapdimensionswecanconsider
thescaleuptowhichacommontopologicmaporganizationcanbeidentiﬁed.NelsonandBower
[208]thereforesuggestthreecategories,continuousmap,patchymap,andscatteredmap;see
Fig.1.2.Thesecategoriespresumablyarisemainlyfromcomputationaleﬃciencyreasonsaswewill
seeinthenextsection.Strictlyspeaking,thelatter,thescatteredmap,isnotamapasitshows
.yographtopnoIntrinsicallytiedtotheconceptofaneuronalmapisthatofreceptiveﬁelds.Areceptiveﬁeldof
asensoryneuronisdeﬁnedastheregionofspaceinwhichthepresenceofastimulusaltersthe
activityoftheneuron.Sincereceptiveﬁeldsofsensorymapsareeasilyaccessibleexperimentally
theyarewellstudied,andthecorrespondingdatashedlightontheunderlyingmapproperties.

ertiesroppMap1.2.1Aswehaveseenjustabove,mapsareusedinmanyareasinthebrain.Thecompellingquestion
[116]thereforeis:Whatisthefunctionaladvantageofneuronalmaps?Thatis,whychooseamap
structureforneuronalprocessing?Here,wefocusonneuronalmapswithaglobaltopography,viz.,
continuousmaps;cf.Fig.1.2.Fromthetheoreticalperspective,thefundamentalargumentisthat,
incontrasttoarbitrarypopulationcodingorclusteredneuronalarchitectures[51],neuronalmaps
ensureatopographicneuronalorganization.
Thisconceptualadvantageofmapsarisesfromcombiningbasictopologicalinformationwith
actualstimulus-evokedresponses.Thatis,incaseofsensorymaps,thepreferreddirectionofthe
andsensoryactivityorgansalloorwsforreceptorsinisterpretinglinkedthewithﬁringtheirproﬁleactualofmapactivity.neuronsInotherwithwrespords,ecttomergingtheirtopneighographbors,y
withviz.,conpreferredtext-depdirectionendent.xiFisortheexample,sameinas(A)illustratedand(B),ininFig.the1.3,contextalthoughofitstheneighbactivitoringyofneuronsneuronitsi
informationMoreover,conthetentmapdiﬀersorganizationdrasticallyfromunderlyingcase(Athe)to(Bneuronal).processingallowsforaneﬃcient

1.ProcessingSensoryInformation

therepresenlikelihotationodinofterprconetintationuouslyofvaaryingﬁringinputpatternsignonalsaasstospatialchasticmap,thatquantitiesis,.thFeorlikelihoinstance,odtoitalloﬁndwsa
sensoryobjectatacertainposition[60,116,131,227,248].Inthecontextofmaps,wecannot
retrieveanythingfromtheﬁringproﬁleFig.1.3(B)butanoisybackgroundwhereasFig.1.3(A)
presentsquitenicelytheprobabilityofﬁndingacorrespondingsensoryobject.Thatis,thelatter
correlateswithapositionalestimatorforsensoryobjects.Whilesensorymapsencodingspatialor
temporalestimatorsarethemostevidentones,maps,especiallymapsinhigherbrainareas,are
notlimitedtospatialortemporaltopology.Theyarefoundtoencodevariousstimuliunderlying
vkindariousoftopmapisologiestopro(reviewvideabyKnsystematicudsenetal.represen[161]).tationSo,ofgettingcomplextostimthepuli.oint,therealpowerofany
sensoryIntheobconjecttextpresenoftedthisabthoevesis,isthecrucial.likelihoBeyodondinthisterpretationissue,ofmapstheproﬁringvideotherpatternevokcomputationaledbya
advandanBowtages.er[208One,]assuggestedwehavethatseendiﬀerentpreviously,categoriesrelates(seetotheFig.1.2)degreesofoftoptopologicologicorderorder.aredueNelsonto
minimizingcomputationaleﬀortsimilartoparallelcomputers.Thegeneralhypothesisisthatmap
overheadorganizationandanreﬂectsoptimalmapusageloadandbalancing.inputThestructureformerinorderandtotheachievlattereaareminimalfoundtocommbeunicationrealized
andneuronallyscatteredbythemap;tcf.hreeFig.kinds1.2.ofConmapstinmeuousnmtionedaps,abforove,instance,namely,areconfoundtinuoustobemap,usedpatchmainlyymap,for
localoperationsintheproblemspacesuchasfeatureextractionandspatialﬁltering.Wehaveto
bearinmindthatcomputationaltasksareperformedmainlybutnotonlybymeansofsynaptic
connections.Chemicalprocessesarealsousedforinformationprocessing,forinstance,toinhibit
acompleteareaofneuronsrepresentingsimilarfeaturesofthestimulus.Thustheconceptofa
mapdoesnotonlyminimizeneuronalwiringeﬀortbutenablesreasonablechemicalcontrol.“Load
balancing”isachievedbylocalmapdistortionsthatreﬂectdiﬀerencesinsensorydensities.For
instance,thereisamuchhigherdensityofsomatosensoryreceptorsinthetonguecomparedtothe
skincorrespinourondingback.sensoryThemapincreasedwheresensorymoreneuronssensibilityencoofdethethetonguesamepishysicalreﬂectedspacebythandistortionsinourofbacthek.
Thewithinpatcahymoremapsglobalseemcontotext.Butadditionallythisisincludestillnonmatter-loofcalopongoingerationsdebate.toTheanalyzeloscatteredcalsensorymapssucdatah
asidenfoundtiﬁesinthecomplexolstimfactoryuliandsystemanswreﬂectersthethequestionnon-topofographicwhatrathernatureofthanthethatinput.ofwhere.Here,thesystem

mapscomputationalofCollection1.2.2Thereisanevenmoreconvincingexplanationfortheomnipresenceofmapsinthebrain.The
realpowerofcomputationalmapscanbeappreciatedonlywhentheinterplayofseveralmaps
isarbitraryconsidered;transfforormationreviewinsee[the161].repreInsenthetationconoftextofinformation.thisthesisTowegiveadeﬁnecfamousomputationalexampleofashoanw
informationisprocessedinasequenceofcomputationalmapsweconsidertheauditorysystem;cf.
ofFig.the1.7.Finferiorrequency-depcolliculusend(ICC).ingtimeInanextdiﬀerencesstep,arethismapcomputediswithintranslatedainmaptoianthespatialcentralmapninucleusthe
theexternalopticnucleustectumof(forthereviewinferiorsee[collicul154,us155,(ICX)161]).whereAlthoughthethevinformationariousisearlyfurthersensoryprocuesjectedinintheto
auditorysystemarehardlyrelatedtospace,intheoptictectum(OT)theauditorymaprepresents
spaceandevenconnectssmoothlytospatialmapsofothermodalities.
beautifulTheOTexampleanditsofwtheell-studiedconnectionbmammalianetweendiﬀerenhomologue,tmaps.thesupAlleriorsensorycolliculussystems(SC)that[261pro],isvidea
minformationultisensoryasinwaellasmap-likepredominanformprotlyjecttomonosensorythelaSC/OTyers[are261];foundcf.andFig.are1.4.shownWithintoproctheessSC/OT,spatial

1.ProcessingSensoryInformation

Figure1.4:Thesuperiorcolliculus(SC).Thesuperiorcolliculusanditsnon-mammalianhomologue,
thethatproopticvidetectum(informationOT),areinwaellmap-likstudeiedform[mid-brain261].stWithinructuresthetoSC/OT,whichallspatialsensorysensorysystemsinformationproject
ofinformationdiﬀerentisusedmonosensoryfordelivmapseringisunamcomputedbiguousandcommotorbinedincommands,tomforultisensoryreﬁningmaps.sensoryThemocomdalities,bined
andSC/OTforarefurthermutuallyprocessingaligned,inwehighercanspbraineakofareasa[166collection,167,of179maps,216that,260].represenBecausetanallalignedmapsinsensorythe
space.Theﬁgureisadaptedfrom[179].

information.Inmultisensorymaps,mostneuronsrespondtomorethanonesensorymodality.In
doingso,theneuronalresponsetomorethanonesensorymodalitycanbecomplex.Incaseofspatial
andtemporalcoincidencewecanobservesuper-additivity,cross-modalenhancement[134,224,261],
orinverseeﬀectiveness[224]whereasincaseofnon-coincidencecross-modalsuppressioncanbe
found[140].Multimodalmapsarewidespreadinthebrain[139].
BoththemultisensorymapsandthemonosensorylayersarefoundtobealignedintheSC/OT
togainauniﬁedmultisensoryrepresentationofsensoryspace[147,261].Thecombinedsensory
informationcanthenbeusedtogeneratedirectionalmotorresponses[166,167,179,216,260].This
isonlypossiblesince,withintheSC,therearenotonlysensorybutalsomotormaps,i.e.,motor
neuronsorganizedinamap-likestructure.Inmotormaps,systematicvariationsofamovement
arerepresentedtopographicallyacrosstheneuronalarray.Directevidenceforthishypothesishas
recentlybeenfoundineye-trackingexperiments[104].

Inacollectionofalignedneuronalmaps,moreover,externalobjectscanbeidentiﬁedbytheir
positionencodedbymeansoftheﬁringpatternindiﬀerentsensorymaps.Thisdoesnotmeanthat
anobjectisadequatelydescribedifonlythepositionisknown.Itratherimpliesthatpositionserves
asfordeﬁningappropriateaandsensory–inobconject;trastcf.toFig.1.5.higherWhenccomputationalombininglevelsdiﬀeren[65t,98]sensory–necessarysystems,theinformationspatial
informationisneededtobindinformationassociatedwiththesamepositioninthemonosensory
mapsintoonesinglemultimodalperceptforfurtherprocessing.

1.ProcessingSensoryInformation

Figure1.5:Objectformationbyspatialinformationbinding.Withinanatlasthereisasimple
waactivitytoydeﬁnepatterna(redsensoryobcolumn).jectbyUptomeansnoofw,sucgroupinghamecspatialhanismisinformation,speculativviz.,e.Hospatialwever,congrueitcannt
bedynamicrelevanent,forvironment.instance,Theretoareselectlinesamofultimoevidencedalsupptargetorting(e.g.therunningexistenceandofsucbarkinghamecdog)hanism.withinAa
andprominenvisualtmapexample[210].aretheTheseneuronalcircuitsare“AND”suppandosedto“OR”idencircuitstifyexternalwithinobthejectsOTsucthathaslinkpreyinfraredand
enhancetheircontext-dependendsaliency.

Bearinginmindtheadvantagesofneuronalmapsthatareduetotopographywenextfacethe
questionofhowaneuronalmapcandevelop.Laterinthisthesis,weadditionallystudyhowa
collectionofmapscanevolve.

developmentMap1.2.3Onemightarguethatneuronalmapssimplyexistbecausetheirneuronalarchitecturefollowsthe
sensorysurfaceoftheirinputmodality.Fromthispointofview,forinstance,thevisuallayersare
retinotopicallyorganizedbecausetheyreceivetheirinputfromtheretina.Similarly,afrequency
mapjustreﬂectsthetonotopicorganizationofthecochlea.
Interestingly,thisargumentdoesnotholdforeverysensorymapasitalreadybecomesapparent
intheexistenceofauditorymapsforinterauraltimeandamplitudediﬀerences.Regardingexamples
inthespatialdomain,letusconsiderthesensoryunitsofthefroglateral-linesystem[295]orthe
snakeinfraredsystem[109].Thesesensoryunitsreceiveacomplexsuperpositionofinputfrom
severaldiﬀerentspatiallocations.Computingamapfromsuchacomplexinput[75,76,95,252]is
certainlynotstraightforward.So,howcanmapsevolve?
Themostobviousideaistoimagineanexplicitgeneticmechanismencodingmaptopography,
viz.,connectivity.Inthiscase,however,weha12vetogenetically15encodealargenumberofpossible
connections.Ourbrainconsistsofabout10neuronsand10synapticconnections.Thatis,
wehavetoencodeanumberofpossibleconnectionscomparabletothenumberofatomsinthe
universe[267].Ofcourse,our2−3∙104humangenescannotachievethisdirectly.Consequently,
intrinsicmechanismsmustdeterminemapdevelopment.
Inthelastyears,neurobiologistsformulatedfourmainhypothesesastohowtopographic-correct
synapticmapconnectionsarise;reviewse.g.[34,181,190,271].First,followingtheaddress
labelhypothesis,synapticstructurespossessaseriesofpositionallabelstellingingrowingaxons
orpre-synapticaxonterminalshowtoarrange.Second,neighboringaxonsconservepre-synaptic
topographyduringthegrowthprocess.Therefore,thepost-synapticstructuresaretopographically
arranged.Third,boththeingrowingaxonsandthetargetstructuresdevelopinatopographically

1.ProcessingSensoryInformation

arrangedtimesequence.Thetemporallymatchingentailsaspatialmaptopography.Fourth,pre-
synapticaxonterminalsarrangethemselvestopologicallybymeansofelectricalactivity.Although
evidenceforeachofthesehypothesescanbefound[271],severalmechanismsseemtocooperate
toensureapropermaptopography.Togiveanexample,electricalactivity-basedadaptionis
supposedtoﬁne-tunemapconnectivityaftersomeimpreciseprewiringwhichcanbebasedonother
mechanismssuchaschemicallabeling[13,146].
Precisemaptopographyiscrucialforsensoryperformance.Inthenextsection,wetherefore
shedlightonactivity-based,viz.,spike-dependentmapformationbyconsideringlocalmechanisms
andglobalstructuresresponsiblefortheplasticityofmaptopography.

1.3Learninginsensorysystems
InThattheis,consensorytextofactivitsensoryysys(input)tems,leadslearningtoadiﬀerenbasicallytmapmeansactivitacyhange(output).intheinpuConsequent-outputtly,learningrelation.
tak1949,esplaceHebbbyalreadymeansofctranslatedhangingthisthegeneralconnectioninsightofstrengthsactivitbetwy-basedeen(inputplasticitandyintooutput)neuronalneurons.termsIn
byformulatinghisfamousrule2[112].Inshort,ifapost-synapticneuronexcitesaspikeshortly
wafterords,oneHebb’sofitslearningpre-synapticruleneudescribronseshasthatﬁred,tempthisoralparticularcorrelationsofconnectionneuronalisstrenactivitygthened.–ifIncausalother–
wareorldlearned.andthTusocutHebb’salongrulestoryimpressivshort,elyneuronalenablesusactivittoyrelatesunderstandtypicallyverytoevgenerallyentsinhothewtoexternallearn
correlationintheexternalworldbymeansofalocalneuronalmechanism.
withIntheHebb’sconruletextofarise.thisFirst,thesiscanw“PredictingeSeunderstandnsoryPaspectserformance”oftsensorywopquestionserformancdirectlyebymeansconnectedof
studyinglocalmechanismsofsynapticplasticity?Second,whatprovokeslearning,i.e,theadaption
ofsensoryabilities?Moreprecise,whichglobalandintrinsicmechanismsassureproperguiding
inputalignedforlocollectioncalofsynapticmapsplasticitfoundyintotheevolvSC?eacomplexbrainstructure,forinstance,suchasthe

mechanismscalLo1.3.1Inthelastdecades,neurobiologicalevidencehasbeenfoundtosupportHebb’srule.Various
mechanismsunderlyinglong-termpotentiation(LTP)andlong-termdepression(LTD)ofsynaptic
transmissionhavebeenstudiedingreatdetailevenifnotyetfullyunderstood;reviewse.g.
185].177,128,34,[13,Inaddition,togainfurtherinsightintothistopic,Hebb’sessence,synapticweightdevelopment
duetocorrelatedactivityofcorrespondingpre-andpost-synapticneurons,hasalsobeenformulated
inmathematicalterms;fordetailedreviewseee.g.[115,145].Thedevelopmentofsynaptic
connectionstrengthJijbetweenapre-synapticneuroniandapost-synapticneuronjcanbe
dividedintothreepartsasillustratedinFig.1.6.First,thearrivalofapre-synapticspikeattime
timinducestheweightJijtochangebyanamountηwpre.Second,iftheconsideredpost-synaptic
neuronjexcitesapost-synapticspikeattnj,allconnectionsarechangedbytheamountηwpost.The
quantityη>0isasmallparameterandtheweightchangeswpreandwpostcaneitherbepositiveor
negative.InFig.1.6weassumedwpre>0andwpost<0.Thethirdpartisduetothespike-timing
ofpreandpostspikes.Giventhetimediﬀerencestim−tnjbetweenpre-andpost-synapticspikes,
JijischangedbytheamountofηW(tim−tnj),wherethelearningwindowW(tim−tnj)denotesa
2gro“WhenwthanproaxoncessoformcelletaAbisolicnearchangeenoughtaktoesexciteplaceincelloBneandorbrepotheatedlycellssucorhperthatsistenA’stlytakeﬃciencyes,partasinoneﬁringoftheit,somecells
ﬁringB,isincreased”;[112]p.62

1.ProcessingSensoryInformation

Figure1.6:Thesynapticlearningprocess[115,145,146].Thediagramillustratesthelearning
processduetothethreepartsofSTDP.Inthiscase,everyspikeofthepre-synapticspiketrain
(toprow)strengthensthesynapticweightJijbywpre(green).Allpost-synapticspikes(second
upperrow)induceanegativeweightchangewpost(orange).Thecontributionofthethirdpart
ofSpike-timing-dependentplasticity(STDP),thecorrelationterm,dependsonthespiketiming
andthelearningwindow(middlerow).Intheleftpart,thepairofpre-postspikesistemporally
toofarseparatedascomparedtothedimensionofthelearningwindow.Theremainingtwoother
possibilitiesaredepictedintherightpart.Ifapre-synapticspikeprecedesapost-synapticspike,
thesynapsestrengthincreases(blue).Incaseofoppositetemporalorder,thesynapseisweakened
(purple).

ofrealFig.valued1.6.Finallyfunction.,weWecanhavecombinedepictedtheatthreeypicalpartsformandofaderivelearningaHebb-twindoypweinthelearningthirdruleuppbasederroonw
spike-timingforatimeinterval[t,t+Δt]
ΔJij=ηwpre+wpost+W(tim−tnj).(1.4)
timtnjtim,tnj
npTheabost-synapticoveequationspikecantrainsaltsopre;pbeost(t)form=ulatedδ(tin−intin;j)tegralandform.computeWethereforedeﬁnethepre-and
ΔJij=ηt+δtdtwpretipre(t)+wposttjpost(t)+t+δtdtW(t−t)tipre(t)tjpost(t).
tt(1.5)Both,correlateequatipre-onand(1.4)pandost-synaptic(1.5),represenactivittya.Inmathematicalparticular,theHebb-tlearypeninglearningruleisrulebasedsinceonthetheequationsprecise
timingsequenceofpre-andpost-synapticspikes.Suchlearningschemesarethereforedenotedas
Spike-timing-dependentplasticity(STDP);fordetailedreviewseee.g.[1,15,55,78,115,145].
Kempter,preGerstner,andvanpHemmenostdemonstrated[145]thatlearningschemesbasedon
pre-synapticνiandpost-synapticνimeanﬁringratessuchas
ddtJij=a0+a1νipre+a2νjpost+a3νipreνjpost+O[(νipre)2]+O[(νipost)2](1.6)
arenotcapabletocatchtheneuronaldynamicsonaplausibletemporallevel.Togiveanexample,
meanratedescriptionsrequirethatallchangesofneuronalactivityareslowatatimescaleofthe
learningwindowW.Thatisnotnecessarilythecase.TheHebbianlearningwindowincortical
areas,forinstance,isabout100msascomparedtothefrequencyof40Hzcorrespondingtoatime
scaleof25msonwhichneuronalactivityvaries.

1.ProcessingSensoryInformation

Figure1.7:Siteforplasticityandoriginofguidinginputformapadaptionintheauditorypathway;
adaptedfrom[78,156].Theauditorysystemprocessestheinitialsensorycues,forinstance,interaural
timediﬀerences(ITD)togainaspatialrepresentationofsoundsourcelocationsintheoptictectum
(OT)thatisalignedtomultimodalmapsintheOT.Inshort,sensorycues,i.e.,frequency-speciﬁc
ITDmaps,arepresentinthecentralnucleusoftheinferiorcolliculus(ICC)andareprojected
totheexternalnucleusoftheinferiorcolliculus(ICX).Fromthere,spatial-auditoryinformation
ismappedtotheoptictectum.Experimentsprovidingabnormalvisualinputdemonstratetwo
importantﬁndings.First,thereisaspeciﬁcsiteforplasticity.Thesynapticconnectionsbetween
ICCandICXchangeinsuchawaythattheICXmapacquiresOTtopography.Second,audition
adaptstoinputfromtheOT.Inresult,visionseemstoguideauditorymapdevelopment[159,160].

ThisshortexampleillustratesnotonlytheadvantagesofSTDPascomparedtorate-based
learningschemesbutalsoreﬂectsanissuealreadymentionedabove.Thewayofneuronalencoding,
forinstanceratecodingversusspikepatternsthatarephase-lockedtotheinputsignals,mayreduce
andreﬁnetheimbeddedinformationandthuscorrespondtoaneuronalﬁlterintheprocessof
neuronalencoding;seeSec.1.1.2.
Inthissectionwehavequantitativelyseenthatpre-synapticinputthatdrivespost-synaptic
activitycaninduceplasticityofcorrespondingconnectionstrengths.Tounderstandandpredict
theconsequencesoflocalsynapticeventstosensoryperformancewehavetoconsidertheglobal
contextwhereinthelocaleventstakeplace.

1.3.2Globalstructures
Inthissection,weintroducesome,butnotmanyimportantthoughtsaboutlearningbeyond
localsynapticevents.Locallearningrulescandescribehowthelocalconnectionstrengthchanges
accordingtothecorrespondingpre-andpost-synapticactivity.However,withoutknowingwhat
“corresponding”meansinthecontextofaspeciﬁcneuronalsetupofasensorysystem,locallearning
rulescannottellushowthesensorysystemdevelopsasawhole.Therefore,wehavetoaskatleast
threequestionsthatwillguideusthroughthesubsequentlines.First,whatdrivespost-synaptic
activitythatresultsinsynapticplasticity?Second,whereinthestructureofpre-andpost-synaptic
synapticconnectionsactivitdoyestoresplasticitultyinappreasonableear?Third,globalhowconnecdotiesons?theHereunpre-synapticto,wetakinputeanmanipulateexemplaryploost-ok
attheauditorypathway.
Wehaveseenabovethatpre-sypaticinputdrivingpost-synapticactivitycanleadtochangesof
thedrivescorrespthepondingost-synapticconnectionactivity?strength.Thatis,Thewhicﬁrsthinpquestionutprobeyjectsondtolothecalpossynaptict-synaptevicentsneuron.is:WhatTo

1.ProcessingSensoryInformation

giveaconcreteexample,aswecanseeinFig.1.7theexternalnucleusoftheinferiorcolliculus
(ICX)receivessensoryinputfromthecentralnucleusoftheinferiorcolliculus(ICC)andtheoptic
tectum(OT).Consequently,theactivityofneuronsinthespatialmapinICXisdrivenbytwo
inputorigins.Verycloselyconnectedtotheﬁrstquestionisanotherone:Whatconstitutesthe
inputfromOT?AsthisistopicofChap.6,weleaveitforthemoment.
Topredicthowtheauditorysystembehaves,wehavetoconsiderthenextquestionofwhere
plasticityappears,i.e.,whichconnectionsarechangedbymeansofactivity-basedlearning?We
knowthatmapsintheOTaremutuallyalignedandthatthisisaresultofsensoryexperience
duringgrowth.Theanswertothewherequestioncanhencetellus,forinstance,whichinputguides
alignmentandwhichconnectionsadapt.Inourparticularexample,weseethatonlytheICC–ICX
connectionsshowplasticityandadapttoinputfromOT.Thelatterinputisthereforedenotedas
guiding;cf.Fig.1.7.ForadetaileddescriptionoftheICC-ICXconnections,considerKnudsenet
[156].erpapal.’sHavingclariﬁedthewhatandwhere,“only”thehowremains.Incaseofourexample,Friedel
andvanHemmen[78]havemathematicallyanalyzeddiﬀerentkindsofguidinginputandtheir
practicalconsequencestothelearningprocess.Theyderived,forinstance,thatonlyinhibitory
inputfromtheOT,calledselectivedisinhibition,canreorganizemaptopography,viz.,theICC–ICX
mappingcanbereorganizedsoastocompensateevenaspatial180◦shiftbetweenguiding(OT)
andsensory(ICC)input.ThiscorrespondstofamousprismglassexperimentsdonebyKnudsen
andothersinthebarnowl[156,159]–performedsimilarlyinotheranimals,suchashamster
[200],cat[278,280],clawedfrog[44],ferret[148,150],andsnake[96].Incontrast,excitatoryinput
leadstoalessprecise,lessstable,andslowerlearningprocesswhichresultsadditionallyinan
ambiguousmapping[78].However,inotherscenariosexcitatoryguidinginputfaithfullydescribes
174].[146,yrealitneurobiological

Insummary,derivingmathematicaldescriptionsoflocalsynapticevents,thatis,makingthem
quantitativelypredictive,isimportantbutnotenoughtogaininsightintothedevelopmentof
brainstructuresandtheirfunctionasawhole.Wehavetoconsidertheglobalenvironmentin
enwhichvironmenthetissynapticevconstitutedentsbappyearmeanstoofpredictneuronalwhatmapstheythatcause.bInelongthetoconsensorytextofsthisystems.thesis,Wesucprohvidane
anewmodelforintrinsicmapdevelopmentinthelastchapterofthiswork.Inthenextchapter,
wewillfocusourattentiononageneralframeworkofoptimalmapformationtounderstandhow
sensoryinformationcanbeprocessedbymeansofmapsandhowtheneuronalconnectivitypattern
predicted.ebcan

2.OptimalMapFormation

Mansollteallessoeinfachwiem¨oglichsehen-
einfacher.nichtaucherabertAlbEinstein

Inthestruggleforsurvivalinacomplexanddynamicenvironment,naturehasdevelopeda
multitudeofsophisticatedsensorysystems.Inordertoexploittheinformationprovidedby
thesesensorysystems,highervertebratesreconstructthespatio-temporalenvironmentfromtheir
sensorymodalities.Asweintroducedinthelastchapter,mostanimalsareabletocompute
aneuronalrepresentationoftheoutsideworldinformofaneuronalmap.Herewepresenta
universalframeworkthatallowstocalculatethespeciﬁclayoutoftheinvolvedneuronalnetwork
bymeansofageneralmathematicalprinciple,viz.,stochasticoptimality.Thatis,givenaknown
physicalsignaltransmissionandrudimentalknowledgeofthedetectionprocess,ourapproach
allowstoestimatethepossibleperformanceandtopredictneuronalpropertiesofbiological
systems.rysenso

ointprtingSta2.1Amousehearsarustlinginthegrass,seessomeleavesmovingand–escapesfromthepredator.
Thus,theperceptionoftheoutsideworldbysensorysystemsandtheconsequenttranslationof
theirresponseintoareliableneuronalrepresentationthatallows,forinstance,directionalmotor
commandsisanessentialconceptforsurviving.
Inthelastchapter,weintroducedtheadvantageousconceptofaneuronalmapforrepresenting
theexternalworld(seeSec.1.2),forinstance,theauditoryspaceintheexamplejustmentioned
intheaboveparagraph.Moreover,wesubdividedthesensoryinformationprocessingintothree
fundamentalsteps.Thesearesignalmapping,sensoryencoding,andneuronaldecoding;seeSec.1.1
andtheexampleinFig.1.1.Inthischapter,wewillnowtranslatetheseconsiderationsintoa
mathematicalframeworkofoptimalmapformation.

delmoMathematical2.2Followingtheideaofmapping,encoding,anddecoding,wewanttounderstandhowtoextract
thestimulusatthebestfromthesensoryresponses.Mathematicallyspeakingwehavetoderive
theinversetransferfunctionl(1.3)thatcanperformanoptimalreconstructionofaparticular
stimulusgiventhesensoryresponsesandcomputationallimitations.Thisinversetransferfunction
canthenbetranslatedintoaneuronalconnectivitypattern(seeSec.1.1).
Thederivationweprovidebelowisbasedonthreereasonablesimpliﬁcations.First,weassume
thatallsensorymapsarepurelymonosensory.Althoughithasbeenquestionedwhetherthereare
mapswithoutinﬂuencefromothersensorysystems[262],ourassumptioncanbejustiﬁedbythe
ﬁndingthatmanyspatialmapsareclearlydominatedbyasinglesensorymodality[280].

OptimalMapFormation

receptorSecond,inputwewithsimplifyitstherespinonse.ternalThatprois,cesseswesetwithintheasptransfereciﬁcsensoryfunctionkreceptor(1.2)btoybeidenthetifyingidentitthey
connectsfunction,toandthehencecorresphondi(1.1)ngconnectssensorydrespirectlyonsestheonastimulusone-to-onecbasis.haracteristics,Althoughi.e.,thethismasensoryynotinputbe
givenforarealisticsetting,theframeworkofoptimalmapformationiswithoutlossofgenerality
btheecauseﬁrstwetransfercanincorpfunctionorateh.theInreturn,ﬁlteringwepropgetaertiessetofoftheequationssecondthattransferaremorefunctionkbunderstandableyredeﬁningand
comparablewithcommontechniques;seee.g.Chap.5.
respTheonseofthirdtheimportansensorytsystem.assumptionThatisthatmeansofalinethatartherelationdetecbtoretwrespeentheonsesstimchangeulusandproptheortionallyreceptor
tothesignalstrength.Forexample,themultipolesofasubmergedmovingobject,i.e.the
signalstrengths,translatelinearlytothedetectablewatervelocitiesatthelateral-lineorgans
[[169251,,75213].,172,Nonlinear136,53],relationcansinbetwprincipleeenstimbeulustreatedandwithdetectorourrespmodelonses,aswe.g.,ell;aseelogariAppthmicendixrespA.2onsefor
details.

2.2.1Deﬁnitionoftheproblem
Aswehaveintroduced,anobjectgeneratesasignalsx(t)varyingintimetandpositionxinthe
externalworld.Thesignalmaybe,forinstance,thetime-dependentsoundpressureataparticular
locationormaydenotethepresenceofedgesormovementataparticularpositionwithinthevisual
ﬁeld.Thesignalinducesaresponseri(t)inasetofNsensorydetectors.Dependingontheproblem
atorparthandofaasingledetectordetectorarrayisucwithhas0a≤spi≤eciﬁcNincantervbaleaofbestcompletefrequsensoryrnciesinorgan,thecocsuchhlea.asIntheleftprinciple,ear,
thedetectorcombinesinformationfrompastsignalswithinthewholesensoryspace.Theresponse
isthereforedescribedby
tri(t)=dxdτsx(τ)hix(t−τ)(2.1)
spaceall−∞xThewheretransferthetrfansferunctionfcanunctionbehi(diﬀerent)tincorpforeacorateshthedetectorphi.ysicsofAuditorysignaltransfertransmissionfunctions,andfordetection.example,
diﬀerincorpbetorateweentherighptositionandleftofear.soundInsourcegeneral,andweearcanwithsafelyrespeassumecttothatthehixhead(t)=0midlineforandlargevaluesthereforeof
|xstate|andofta.Thissensor.reﬂectsWewillourinneedtuitionthispropthatertevyentslateroccuon.ringMoreofaravwer,ayorsincelonganyagodetectorwillnotcaninﬂuenceonlyreactthe
xtofunctiontemp(2.1)oral-causal,withi.e.,adaptedpastinsignalstegrationwesetlimitshi(tas)a=0conforvtolution<0.Wwithercanespthenecttorewritetime,theresponse
ri(t)=dx∞dτsx(τ)hix(t−τ)=:dx(sxhix)(t).(2.2)
−∞theTheabodetectorverespequationonseisdescriblimitedesthebyrespatonseleastofthreeanidealfactors.system.Inbiologicalsystemsthequalityof
system.First,Second,informationnoisemayinﬂuencesgetlostalldurinstepsginthethetransferdetectionfromandtheoutsidereconstructionobjecttoprothecess[inside71].Finalsensoryly,
limitationsoftheneuronalhardware,forinstance,thelimiteddynamicrangeofreceptors,constrain
possiblesolutions;seeSec.2.4fordetails.
additionalWithinournoiseterms.mathematicalAccordinglymodel,waetermincorpdesoratecribingthesebacthreekgroundrestrictivnoiseeξx(tfactors)mbustybinetroaddedducintog

FMapOptimalrmationo

Figure2.1:Physicalmapping:signalsx(t)withbackgroundnoiseξx(t)ismappedontoanoisy
receptorresponseri(t)+χi(t)throughthenoisytransferfunctionhix(t)+ηix(t).Optimalmap
formation:the(possiblynoisy)inversetransferfunctionlix(t)+λix(t)givesanestimatesˆx(t)ofthe
signal.Signalsx(t)+ξx(t)
Transferfunctionhix(t)+ηix(t)
Receptorresponseri(t)+χi(t)
Inversetransferfunctionlix(t)+λix(t)
Estimatedsignalsˆx(t)
Table2.1:Functionsanderrortermsdescribingdetectionandprocessingofsensoryinformation.

theadditionalsignal.Fnoiseteurthermore,rmsηix(wt)eandassumeχi(t),thatresptransferectively.functionConsequeandntly,sensory(2.2)isrespmoonsediﬁedaresohampastoeredreadby
ri(t)=dx[(sx+ξx)(hix+ηix)](t)+χi(t).(2.3)
Toreconstructtheestimatedsignalfromthedetectorresponsesri(t),theabovetransformation
musttransferbe“invfunctionsertedl”x(int)bsomeetweenappropridetectorateiwayand.Wtheethemaprefoatrepositcalculateionx.theWhentime-depapplyingendlexn(tt)intoversethe
iireceptorresponsesati,weobtainaccordinglyto(1.3)theestimate
sˆx(t)=[ri(lix+λix)](t)(2.4)
ioftheoriginalsignalsx(t).Herethehatonsˆx(t)denotesareconstructionandthetermλix(t)
noterepresenthattsintheconnoisetrastduetoeltostheewhere[concrete217,229]realizationthepreofsentthemodeltheoreticalisinvnon-iterativersee.transferThiswillfunction.resultWine
apurelyfeedforwardnetworkstructurewhenitcomestoaneuronalrealizationinSec.2.4.
Figure2.1illustratesthewholemathematicalprocedureofsensoryinformationprocessingas
describsection,edweinwillthesteindicatepsofhoFig.wto1.1.Allcalculatetheinvreleverseanttransfertermsaresfunctionsumlixmarized(t)thatinTenableable2.1.optimalInthesignalnext
reconstruction.2.2.2Optimalreconstruction
Wewanttotuneoursensorysystemtooptimallyreconstructnotonlyonespeciﬁcsituationbut
thethatwtypicealdenoteenasvironmen“tt.ypical”.InotherwConsequenords,tly,abiologicallyspeciﬁcrelevsensoryantsignsignalsalisbaelongconcretetoaclassofrealizationsignalsof
aminimizeclassofttheypical,expectationbiologicallyvalueofrelevtheantsquaredsignals.Thatdiﬀerenceis,bitetiswaeenstocsignalhasticandquanretity.constructionWe.therefore
ofphThisysicalispossiblemappingbandecausetheallneuronalquantitiesprocesandsoffunctionsoptimal(seemapFig.formation2.1)inv(Secolv.ed2.4)inbareoth,self-athevproeragingcess
aswedemonstratein[35].Themathematicaldeﬁnitionofself-averagingallowsforadescriptionin
termsofexpectationvalues.

OptimalMapFormation

Toderivetheinversetransferfunctionslix(t)thatenableoptimalsignalreconstructionfora
classoftypicalsignals,wecannextminimizetheexpectationvalueofthesquarederrorbetween
signalrealandestimatedE{lx(t),t}:=dtdx[sx(t)−sˆx(t)]2=dtdx[sx(t)−sˆx(t)]2.(2.5)
tt
t−Tt−T
Herethebrackets.denotetheexpectationvaluewithrespecttothediﬀerenttypesofnoise,and
Tisatypicalprocessingtime.
Tobemathematicallyprecise,anexpectationvalueisanintegralonaprobabilityspacewith
respecttoaprobabilitymeasurep.Forarbitraryfunctionsfandg,if|f−g|2=0thenf=g
withrespecttopor,physically,lookingattheworldthroughp’sglasses:whatpﬁndsimportant
popsupclearlywhereaswhatpﬁnds“irrelevant”hashardlyanyweight.Thelatterneednot
correspondtowhatwe“think”ourselves;seevanderWaerden[276].
Aquadraticformoftheerrortermhasbeenproventobeareasonableandpracticalchoicein
manyphysicaloptimizingproblems;see[193].IncaseofindependentGaussianerrorterms,the
formulationviaaquadraticerrorisundercertainconditionsidenticaltoresultsobtainedbymeans
ofmaximum-likelihoodestimates[135,144];seeSec.2.3.2.
Mathematically,theerror(2.5)isafunctionalassigningtoeverysetofinversetransferfunctions
onespeciﬁcvalue.Minimizationoffunctionalsintheaboveintegralformisacentralandwell-
studiedaspectofthecalculusofvariations[45,85,138,32].Forthepresentsituation,theﬁrst
variationwithrespecttoeveryinversetransferfunctionlj(x,t)istovanish.Thatis,
∂[sx(t)−sˆx(t)]2
∂ljx(t)=0foreveryj.(2.6)
Inordertosolve(2.6),wehavetosubstitute(2.4)fortheestimatesˆx(t)andreplaceri(t)byits
description(2.3).Expandingthesquare,weencounterexpectationvaluesofproductsconsistingof
varyingcombinationsofnoiseandsignalterms.Hereweassumethatallnoisetermsaswellasthe
signalitselfarestochasticallyindependentofeachothersothattheexpectationofaproductof
independenttermfactorizes;forinstance,
sx(t)ηix(t)=sx(t)ηix(t).(2.7)

Foraproductconsistingofthexsamekindoftermweneedtoconsiderthedeﬁnitionofthe
autocorrelationofaquantityf(t)asgivenby
fx(t)fx(t)=δ(x−x)δ(t−t)(µf2+σf2)(2.8)
withµfthemeanandσfthevarianceofthequantityfx(t).Thatis,weassumeinaﬁrststepthat
thevaluesfordiﬀerentspatio-temporalpositionsarecompletelyuncorrelated,akindofworst-case
analysis.Sincethemeansofallnoisetermsµfvanishwegetthefollowingcorrelationterms
ξx(t)ξx(t)=δ(x−x)δ(t−t)σξ2,(2.9a)
χi(t)χj(t)=δijδ(t−t)σχ2,(2.9b)
ηix(t)ηjx(t)=δijδ(x−x)δ(t−t)ση2(2.9c)
with|x|<xmaxand0<t<tmax.(2.9d)

rmationoFMapOptimal

Throughtheﬁnalequationwetakeintoaccountthatthenoiseηix(t)vanishesforlargevaluesoft
and|x|,inthesamewayasforthetransferfunctionhix(t).
Theautocorrelation(2.8)ofthesignalsx(t)itselfdependsontheproblemathand.Eitherthe
detectorsofthesensorysystemmeasureabsolutesignalstrengths(µS),e.g.,vision,ormodulations
ofameanvalueofthesignal(deviationσS),e.g.,audition.Inanycase,onehastochoosethe
correspondingbiologicallyrelevanttermandputtheotherequaltozero.Inthefollowing,we
choosetheexpectationvalueµS2ofthesignalastheappropriatequantityandthereforetakeσS2to
zero,ebsx(t)sx(t)=δ(x−x)δ(t−t)µs2.(2.10)
While(2.9)incorporatesreasonableassumptionsforallnoiseterms,thecorrelation(2.10)forthe
signalisastronghypothesis.Signalsarenamelycharacterizedbyspatio-temporalcontinuity;e.g.,
objectsandtheircorrespondingsignalsusuallydonotdisappearfromonepointintimetothenext.
termcorrelationGaussianAsx(t)sx(t)=Aexp−|x−x|2/(2σx2)exp−|t−t|2/(2σt2),(2.11)
forinstance,cantakeintoaccountcorrelationsbetweenneighboringpointsinspaceandtime.Here
σxandσtaretypicalspatialandtemporalcorrelationscales.TheapplicationofsuchaGaussian
correlation,however,doesnotgreatlyalterthefurtherderivationbutonlysmoothenstheﬁnal
estimatedsignal.Forreasonsofclarity,wewillthereforesticktotherelation(2.10);seeAppendix
details.for[35]andA.4Returningto(2.6)wehavetosolveitandinsodoingapplythecorrelations(2.9)and(2.10)so
astoarriveat
maxljx(t)σχ2+(µs2+σξ2)|y|<ymaxdydτση2+(µs2+σξ2)dy(hiylix)◦hjy(−t)=µ2shjx(−t)
i<t<τ0(2.12)fordetailsseeAppendixA.3.Theopencircle◦denotestheautocorrelationintegral
∞(a◦b)(t):=dτa(τ)b(t+τ).(2.13)
−∞Inordertosimplify(2.12),wedeﬁnetwonewnoisemeasures,
2τ2:=σ2ξ(2.14)
µsandσ2:=σχ2+dydτση2(µs2+σξ2).(2.15)
µs20|y<τ|<y<tmaxmaxµs2
Theparameterτrepresentsaninversesignal-to-noiseratio.Therefore,itisoftenreasonableto
assumeasmallvalueofτ.Theparameterσ,ontheotherhand,describestheoverallmeasurement
noisebyrelatingdetectionandtransmissionnoise,σχandση,tothesignalmeanamplitudeµs.A
priori,itsvaluecannotbeassumedtobesmallandhastobeadjustedaccordingtothesituationat
hand.Inordertofurthersimplify(2.12)weswitchtoFourierspace,whereconvolution(2.2)and
correlation(2.13)becomeordinarymultiplicationscombinedwithcomplexconjugations.Using
(2.14)and(2.15)anddenotingFouriertransformsbycapitallettersandthecomplexconjugation
byanoverline,(2.12)simpliﬁesto
Lixσ2δij+(1+τ2)dyHiyHjy=Hjx.(2.16)
i

OptimalMapFormation

Equation(2.16)isthemainresultofourderivation.Inprinciple,itallowsustocalculatethe
x–invseeerse(A.16)transferinthefunctionsAppLendixiffororoptimaldetails–signalthenconﬁrmsreconstruction.thatAthecalcinvuerselationofthetransformationsecondvweariationhave
foundindeedminimizestheerror.Forconveniencewewillintroduceanalternativenotationinthe
section.nextionnotatMatrix2.2.3Torewrite(2.16)inamorepracticalnotationweintroduce“matrices”HandLbyputting
H[ix]=Hix,L[xi]=Lix.(2.17)
Thenotationsillustratethattransferfunctionsandinversetransferfunctionsarelineartransforma-
tionsfromacontinuousspace(theoutsideworld)intoadiscretespace(theneuronalmap)and
viceversa.Therefore,HandLareonlyformallymatriceswithaspatialcoordinatexvaryingin
R.Thematrixmultiplicationinvolvingthespatialcoordinatemustconsequentlybeunderstood
asanintegration.Adiscretizationofspace,asisusualinnumerics,wouldleadtoatruematrix
ulation.formInaddition,weintroducethecovariancematrixC(R)ofthereceptorresponseRasdescribed,
e.g.,in[135,144].Inourcaseweﬁnd
C(R):=(R−R)(R−R)T=µs2σ21+τ2HHT(2.18)
wherethesuperscriptTdenotesthematrixtransposeand1theidentitymatrix.Equation(2.16)
nowsimpliﬁesto
MLT=HwithM:=µs−2C+HHT.(2.19)
GivenMasaninvertiblematrix,denotedasthe‘modelmatrix’,thesolutionforLturnsouttobe
sL=M−1HT=HTµ−2C+HHT−1.(2.20)
Thisequationgivesauniquesolutionfortheoptimalreconstructionforanygivensetoftransfer
functionsandnoiseconstants(σ,τ).Using(2.4)inmatrixformweﬁnd
Sˆ=LR(2.21)
asestimatedsignalfromthemeasuredresponsevectorR.
2.3Relationtocommonmethods
Thechallengeofsignalreconstructionhasalongtradition,and,accordingly,onemayaskhow
theaboveformalismrelatestomethodsthathavebeenestablishedinthisﬁeld.Inthefollowing,
wewilldiscusstherelationofourmodeltomethodsbasedonthepseudo-inverseandtothe
maximum-likelihoodapproach.
Pseudo-inverse.2.3.1Ifthenoisetermscanbeneglected,theinterpretationof(2.19)isstraightforward.Inthiscasethe
covariancematrixCvanishesandtheresultingequationleadsto
L=HT∙H∙HT−1.(2.22)
Thepseudo-inversereconstruction1ofH[12functions].AtLthathindsighwet,havethisjustmakesfoundsensefulﬁllsincetheptheropertiespseudo-inofvtheerseMoore-Pgeneratesenrosean
1TheexplicitexpressionforLin(2.22)onlyholdsif(H∙HT)isinvertible.

rmationoFMapOptimalinvapproersionximatemayinvnotersebepmatrixossiblethatforaminimizesmatrixHthethatquais,dratice.g.,error;rectangularseeAppinsteadendixofA.5.squareAnorexactof
Butincompleteevenrank.inthemoregeneralsituationofnon-vanishingnoiseterms,wecanobservestrong
therelationscalculationbetweofentheourframewpseudo-inorkverse,andamethodsregularizationbasedonhasthtoebeinpseudo-introvducederse.toThesuppresspointisnoisethat,terms,for
ttermypicallyα1tomakehigh-frequencyitmorevstableariations.[193,The268,269,so-called228].TheTikhonov-Milregularizedlerreequationgularizationthenaddsreadsapositive
α1+H∙HT∙LT=H.(2.23)
2Hence,Comparinginthisthispsecialequationcaseourwithgeneral(2.19)weapproacseehthatisαidencorrespticaltoondsmethoexactlydsusingtoourthetermTikhonoσifτv-Miller=0.
regularization.2.3.2Maximum-likelihoodapproach.
Thedata.Withinmaximum-likelihothemaximodum-lanalysisikeliho[o135d,sc144heme]isaonecommoncomputestooltheinthestiminulusthatterpretationistheofmostlikmeasuremenelyonet
givenasetofdetectorresponsesR.Experimentshaveshownthatoptimalornear-optimalstimulus
Acommethobinationsdofopticanmalindeedstimulusdescribecomseveralbinationsucphenomenahasofthesensorymaximproum-likcessingeliho[o67d,126approac,165h,is2,113,therefore202].
highlyrelevanttoneuronalinformationprocessingandoughttobeincludedintoourmodel;see
1.Chap.Themaximum-likelihoodapproachtriestoﬁndthemostprobableinputsignalSgiventhe
(σdetector=∞).respWenoonseswR,assumeaknoawnlineartransferrelationfunctionH,andnoaprioriknowledgeaboutthesignal
sR=HS+χ(2.24)
withχrepresentingthenoise.WeassumethenoisetofollowaGaussiandistributionwithzero
meanTheandmethothedstandardminimizesthedeviationnoiseσχ.χ.Thatis,basedonthefundamentaldeﬁnitionsofBayesian
statistics,itmaximizestheconditionalprobabilitydensityfunction
p(R|S)∝exp−2σ1χ2(R−HS)T(R−HS)(2.25)
withrespecttothesignalS.Thisleadstoalinearsystemofequations
S=HTH−1HTR.(2.26)
=:LML
Usingtheaboveassumptionsforourmodel,viz.,σs=∞,η=0,andξ=0,(2.19)reducesto
HHTLT=H.(2.27)
Totestwhetherthetwoﬁltersareequal,weinsertLMLinto(2.27).Applicationofthetransposition
rulesshowsthatwiththeassumptionsweusedL=LML,thetwostrategiesareidentical;for
lodetailscalizationwerefercapabilittoyofelsewherethe[135,lateral-line144,242sys,te236m,].forWewillinstance,useofthisﬁshminethothednext(2.25)ctohapter.quantifythe
19

OptimalMapFormation

Figureconnects2.2:tosevNeuronaleralmaprealizationneurons.ofTheunimomapdalneuronsmap(encoformation.dingtheEachlocationsensorx)(heremayhairreceivcellse(mlabeledultiple)i)
thiswconnectionsay,thefromeactransformationhsensor.lix(Eact)hcanbeconnectionreliablyhasawrepresenell-deﬁnedtedinastrengthneuronalandnetwtemporkoral[76].delayt.In

2.4Neuronalrealizationofthemodel
Inthissectionwerelatethegeneralmathematicalalgorithmofoptimalstimulusreconstructionto
ainSec.concrete2.2.2neuareronalfulﬁlledconintext.Wneuronalethereforeprochaessing.vetoThatverifyis,wﬁrsteneedwhethertochecthekwhethassumptionsertheweneuronalmade
quantitiesandfunctionsofoptimalmapformationareself-averaging.Tothisendwenoteonthe
onedescribhandedbthatyastoﬁrincghasofticproneuronscess,isforcorrelatedinstance,withaGaussianneuronalone;inputweandwillthatseeinneuaronalminutenoisewhy.canOurbe
frameworkcancopewithanydistributionofneuronalxnoiseaslongasthemeaniszero.Onthe
theothermapshandassotheciatedoptimalwithinvdiﬀerenersetmotransferdalitiesfunctionsandlihence(t)arereﬂectlearnepropdertiessynapticoftheconnectionsunderlyingbetlearnweening
protimecess.scalesforEﬀectivelearninglearningandissloindividualwbecauserealizationsitneedsofanmanyexternalindepsignalendentcanrepbesetitions.eparated.AccordInotheringly,
ofwtheords,prolearningcess;seeis[a145self-a].Asvmeneragingtionedprocessbefore,wherequanonlytitiesaverandagedfuquannctionstitieswithinenterthebyphtheysicalverymappingnature
proexploitcesstheareself-amathematicalveragingasframewwell;orkaspleasederivseeedalinso[Sec.35].2.2Inareconclusion,fulﬁlled.theconditionsneededto
xInsuchConsequenanarctly,whitecture,ecanthenowactualtranslateprothecessinginveisrseptransfererformedbyfunctionstheli(synaptict)intoconnectionsneuronalbhardwetweenare.
neuronsanddetectors.Spatialprocessingisgovernedbythetopographicstructureofthenetwork;
thatdeterminedis,whicbyhthedetectordistributionisconnectedofdelatysowhicwithinhtheneuron.setofTempconnections.oralproFigurecessing2.2onshothewsotheranhandexampleis
ofsuchaneuronalsetup.
andInthetheabensuingovederivmapationthroughweahavediscretealreadynumtakberenofintoinverseaccounttransferthefdiscreteunctions.charFacurthtereofrmore,detectorsthe
discrete,“spiky”characterofresponseandreconstructionbytheneuronalrealizationisalready
xthetakeninvcareerseofbtransferythefunnoisectionstermslx(tχ).iandThisλi.Thatdiscretizationis,weisarerealizedleftwithbyathesamplingtemporalprodiscrcedureetizationwhereofa
ixbnumeenbershoofwnthatdendritesawithlimitedapnumpropriateberofdelayssynapticischosenconnectionstorepresensuﬃcestthetocompletesamplelithe(t).timeIthascourseindeedof
lix(t)[76].Evenmoreso,themap-neuronresponseisrobustwithrespecttothesamplingmethod
ofthetemporaldelays[178]aswell.

FMapOptimalrmationo

asimpleConsequenfeedforwtly,ardasnetwillustratedorkofinFig.excitatory2.2,ouranduniﬁedinhibitoryframeworkconnectionscanbineorderimplementoformtedbayunimomeansdalof
mappatternfromisarbitraryestablishedininpuat[real76,252,biological178].Itsystem.doesHerenot,thehowevcorrecter,explainsynaptichowconnecsuchtaionshavconnectivitetobye
learned.Ithasbeenshown[74,78]thatateachersuchasthevisualsystemcangeneratecorrect
synapticstrengthssothatamapcanindeeddevelopinothermodalitiesbymeansof(supervised)
STDP;compareforthedetailslearnedseeSec.connectivit1.2.3,ypSec.attern2.5.3,withandtheChap.optimal6.oneThanksastogiventhebypresen(2.16)tandmethod(2.20).wecan
Abiologicalmeansetup,ingfulthough,comparisonmayofnotthebestraighmathematicallytforward.Inoptimalrealnetwbiologiorkcalarcsystems,hitectureerrorwithanreductionactual
asin(2.5)toitsminimum–thatis,realizingtheoptimalconnectivity–maynotbepossible
becauseofneuronallimitations.Thelimitedneuronalaccuracythatresultscanbeincludedinto
ourframeworkbyreducingtheerroronlybelowacertainerrorthreshold,whichmayevenvaryin
space.Forinstance,thesamplingarraysofanimaleyesarenon-uniform,withdiﬀerentpartsof
fothecusonvisualspﬁeldeciﬁcbeingspatio-tempsampledoralwithdomainsdiﬀerentcanspatialmathematicallyandspectralbererealizedsolutionby[in125tro,257ducing,293a].pSucositivhea
wcertaineightingfuthreshold,nctiontheintoareathesinwithintegraltheoffo(2.5).cusoftheAccordinweighglyt,whenfunctionhareducingvetothereachglobalahighererrorbleveloelwofa
optimization,i.e.,ofresolution,thantherest.
AsindicatedinSec.1.2,theconceptofreceptiveﬁeldsisincludedinourformalism.Itmaybe
goodtoremember,though,thattherearetwomappingfunctions(LH)andLprojectingdirectly
ontothemap.Sincetherowsofthesemappingfunctionscontaintheinformationfromwhichareas
aofspreceptiveciﬁcemapﬁeldsneuronwhenwreceiveescomeintoput,theauditoryrowsdessignalcribproethecessingreceptivineChap.ﬁelds.3.WTheere,willweusetaktheeaconceptcloser
lookattheneuronalimplementationbymeansofaconcretesettingandexploittheconnection
betweenoptimalmapformationandreceptiveﬁelds.
Takentogether,theformalismofoptimalmapformationiscapabletodeliveranoptimal
neuronalconnectivitypattern,justasillustratedinFig.2.2,andherebydirectlygivesaforecastof
howexampletheinthereceptivconetextﬁeldsofaretheshapvisualed.Insystemthetonextillustratesection,theweprocapabilitiesvideavoferythesimplepresentbutframewconcreteork.

Spatialexample:visualprocessing
Withinthevisualsystemeachsensoryneuronisbasicallytunedtoaparticularspatialposition.In
mathematicalterms,everyretinalneuronireceivesinputfromaspatialpositionxi,itspreferred
position,andneighboringpositionswithinaregiondeterminedbyresolutionρ.Thetransfer
functioncorrespondingtosuchasensorysystemis
2hix(t)=exp−|x−xi|δ(t),(2.28)
2ρ2

anditsFouriertransformreads

(2.28)

2Hix=exp−|x−x2i|.(2.29)
ρ2Withinourexemplarysetupweassumethatthesignalpositionx=(u,v)encodespositions
u,v∈[−1/2,1/2].Asaremind2er,wehaverescaledpositionssoastomakethemdimensionless
andﬁtinthesquare[−1/2,1/2].Fromtheaboveansatz(2.28)and(2.16)wecalculatethematrix

OptimalMapFormation

Figure2.3:Spatialreceptiveﬁeld.Connectionstrengthstoamapneuronencodingtheposition
(u,v)=(0.1,−0.2).Thesensoryneuronsaredistributedona40×40gridwithpreferredpositions
u,v∈[−1/2,1/2]andatuningcurvewidthρ=0.9.Wechoseσ=1andτ=0.Aclearcenter-
surroundreceptiveﬁeldemerges.Receptorneuronsthathaveapreferredpositionmatchingthat
ofthemapneuronhaveexcitatoryconnections(whitespot).Receptorneuronshavingaslightly
oﬀ-setpositioninhibitthemapneuron(darkcircle).Neuronswithpreferredpositionsfaraway
fromthemapneuronhaveconnectionstrengthzero(gray).

tsonencomp2
Mij=σ2δij+(1+τ2)exp−|xi4−ρ2xj|×erfui+2uρj−1−erfui+2uρj+1
(2.30)vi+vj−1vi+vj+1
×erf2ρ−erf2ρ
whereerf(x):=√2π0xexp−y2dyistheerrorfunction.Toﬁndtheconnectionstrengthslix,
wenumericallycalculatethemodelmatrixMforadiscretizedspaceandparametersσ=1and
τ=0.transformationWiththewematrixcannMumerwiceallythenobtaindeterminelxfortheeachmapconnectionpositionstrengthxassshoL.wnByinanFig.inv2.3.erseFHereourithere
iconnectionsfromallreceptorstoamapneuroni,i.e.,itsreceptiveﬁeld,areplottedforanarbitrary
strongpreferredpropjectionsositiontoxithe=(0.map1,−0.neuron2).(brighClearlyt,sptheotinFig.receptors2.3)encobut,dingintheterestinglypreferred,thepreceptorsositionhathatve
encodeslightlydiﬀeringlocationscontributenegatively(darkcircleinFig.2.3).
Suchacenter-surroundproﬁleiscalled“Mexicanhat”andis,e.g.,realizedbylateralinhibition,
awell-knownphenomenonﬁrstdescribedbyMach[184]inthevisualsystemin1866.Uptonow
forthismecinstance,hanism,insescttudiedvisionin[the130],snakmammalianeinfraredvisualvisionsystem[255[,281252,],143],electrichasbeenﬁelddiscodetectionveredinaswellelectricin,
ﬁsh[250],andsurfacewavedetectioninthebackswimmer[203].
surroundIncontrastreceptivtoemanﬁeldyismoadelsnaturalsuchasconsequencepop-outof[153our]moordelsaliencyandthusdetectionexplains[129,lateral175]ainhibitioncenter-
asandoptimalapplyforourconceptmap-formationofoptimalpurpmaposes.Informationthefollotowingelabcoratelyhapter,wderiveevisanitarcthehetypauditoryemodelsystemfor
signalenhancementandechosuppression.IntheAppendixA.1,weadditionallyprovideaneasy
step-by-step“recipe”soastoﬁndtheoptimalconnectivityinanarbitrarybiologicalsetup.
Next,weleavetheframeworkofoptimalmapformationforthetimebeinguntilweconsider
thethesensoryinfraredpsystemerformanceofsnakofesthe(Chap.auditory5)sandystemrevisit(Chap.general3),asptheectsoflateral-linecross-mapsystem(Chapcomputation.4),thatand
followfromcombiningmono-sensorymaps.

MapOptimalrmationoF

interactiondalMultimo2.5Inthelastsection,wegainedadeeperunderstandingofmonosensorymapsandtheircomputational
power.Thatis,wecanrevisitatopicofSec.1.2,Cross-mapcomputation,andtakeacloserlookat
thisimportantissue.Onthelevelofmaps,wegenerallydistinguishtwocategoriesofmultimodal
interaction,viz.,cross-mapcomputation:integrationandpoolingofinformation.

Integration2.5.1Congruentspatialinformationfromdiﬀerentsensorysystemscanbeintegratedintoasinglemerged
and,hence,multimodalmap.Suchanintegratedmap,ascomparedtounimodalinformation
processing,featuresincreasedinformationreliabilityandsaliencyaswellasanimprovedsensitivity
inbothspaceandtime[166,167,104,237].Forexample,ifvisualandauditorysensorysystems
bothregisterasignal,e.g.,“brownahead”and“barkingahead”,itisveryprobablethatthesignal
correspondstoanactualobjectratherthantoasensoryartefact.Atthesametime,theintegrated
signalwillbestrongerandallowsforfasterreactions(e.g.,“escape!”).Insomecasesanintegrated
signalisevenoptimal[102,202].
Moregeneralneuronalmodelsdescribingmultimodalintegrationandbasedonstatistical
methodshavebeenpresentedelsewhere[59,60,61].Concretetheoreticalmodelsofmultimodal
integrationwithintheSChavebeendevelopedaswell[7,220,238,239,186,274].

olingoP2.5.2Notonlycanthemonosensorymapsbemergedintoamorereliablemultisensorymap,butthe
diverseinformation,thus,signalcharacteristicswithinthemonosensorymapscanbeaccessed
simultaneouslyaswell;rememberFig.1.5.Thissimultaneousaccessingisonlypossiblesince
allmonosensorymapsarealignedandconsequentlyspace-timecanservetolinkthediﬀerent
modalities.Consequently,anobjectatonespeciﬁcpositioncanbeidentiﬁedandcharacterizedin
ordertoselectmotorresponsesinacomplexenvironment.Forexample,arattlesnakemaydetect
spatialcoherentactivityinitsvisualand/orinfraredmap.Onlyiftheencodedobjectisvisible
andwarmitwillbeidentiﬁedasalivingpreyobject.Ifitisvisibleandnotwarmthesnakewill
discardneuronaltheANDandinformation.ORproExpcessingerimenstepstalforevidencethecomforsucbinationhapofoolingvisualofandinformationinfraredismappro[210vided,211by].
TheseprominentexamplesofpoolingintheSC/OTcouldenabletargetselectionandthusensure
appropriatemotorcommandsinacomplexenvironment.
Despiteincreasedreliabilityofanintegratedmap,itsindividualinputstreamscannotbe
distinguishedanymore.Thatis,theinformationaboutwhichmonosensorymaphasdetermined
thepositionislost.Withintheaboveexamplethemultimodalmapmayindicateamultimodal
eventahead,butthetriggeringmodality,thatis,visual,auditory,oryetanothermodalityremains
ed.unresolvInsummary,integrationofinformationallowsforareliablespatialdeterminationofanobject,
thekeytaskofobjectformation.Ontheotherhand,poolingofinformationassuresanaccesstothe
detailscorrespofondsantoobjaectswitchnecessarybetweenforobjeparallelctandidentiﬁcserialation.dataSwitcprohingcessingbettowbeenestinﬁttegrationdiﬀerentandtasks.pooling
Toenableeﬃcientmultimodalinteractionsuchasintegrationandpooling,alignmentofthe
diﬀerentmono-andmultisensorymapsisofcrucialimportance.Onlythencanamultimodal
stimulusataspeciﬁcspatiallocationbeidentiﬁed.Analignmentofsensorymaps,however,isnot
presentatbirthandmustbelearned[116,262,156],asdiscussedinthenextsection.

OptimalMapFormation

Development2.5.3Wenextcomebacktothequestionofhowsensorymapscanbealigned,i.e.,howacommon
multimodalspacecanevolvethatwehavebrieﬂyintroducedinSec.1.2andSec.1.3.Anobvious
solutiontosuchanalignmentprocesswouldbetheexistenceofonedominantmodalityasreference
forallothermodalities[158,156].Thisreferencemapwouldthenautomaticallyleadtomodiﬁcations
maps.otherallofAndindeed,experimentalandphysiologicalstudieshaveshownthat,inmanyanimals,destruc-
tionordisturbanceofthevisualpathwayleadstodisorganizedandabnormalsensorymapsin
non-visualmodalities.Theseﬁndingshavebeenobtainedinhamster[200],cat[280,278],clawed
frog[44],ferret[150],barnowl[159],andinsnakes[96].Psychophysicalexperimentswithcongeni-
tallyblindandnormallysightedhumanshaveshownthatvisualinputearlyinlifeisnecessaryfor
multimodalinteractiontooccur[122,230,233].Consequently,visionseemstoserveasa“teacher”
dalities.monon-visualforAplausibleargumentsupportingtheideaofvisionasteacherinputistheintrinsictopographic
orderoftheretina.Itisknownthatlayersofneuronscanself-organizeintotopographicmaps,
proreadervidedisthatreferredtoinitially[271a].smThisallsettogetherofcorrecwithtlytheorganizedsubsequentneuronsdevelopmenexists[t284of].layFeorrsainthereviewvisualthe
cortex(formice,see[133])mayallowtheintrinsictopographyoftheretinatostep-by-stepdictate
theorganizationandalignmentofhighervisualand,potentially,alsomultimodalmaps.
Thegeneralmechanismfacilitatingtheprecisionofsuchanalignmentofmapsisspike-timing-
dependentplasticity(STDP)[16,17,55,87,115,145,188,254,294].Anexamplewherethe
alignmenthasbeenstudiedindetail,bothexperimentallyandtheoretically,isaudio-visualintegra-
tionwithintheOTofthebarnowl.Hereexperiments[127,158]haveshownthattheauditorymap
followssystematicchangeswithinthevisualinput.Althoughtheprecisenatureofthisteaching
signalhasnotbeenclariﬁedexperimentally,selectiveneuronaldisinhibition,orgating,seemsto
playakeyrole[103,285].Althoughexcitatoryandinhibitoryteachinginputcantheoretically
accountforpropermapalignment[78,56]onlyinhibitoryteachinginputisabletore-alignalready
[78].connectionsmapestablishedInsummary,theabovestudiessupporttheideaofvisionasteachermodalitytoalignother
monosensorymaps,buttherearecontradictingﬁndingsaswell.Wecansummarizetheseﬁndings
intotwomajorpoints.First,visionisnotneededatallasteacherinputforthelearningprocess
ofsensorymaps.Second,visionshowsplasticityasitisinﬂuencedbyothermodalitiesandasit
improvesduringdevelopment.Concerningtheﬁrstpoint,ithasbeenshownboththeoreticallyand
experimentallythat,althoughimprecise,amapofazimuthalsoundlocationcanbelearnedwithout
anyvisualinput[146,158]thoughadmittedlyonageneticallydeterminedsubstrate.Inaddition,
non-visualmodalitiescaninﬂuenceeachotheraswell,e.g.,auditioncaninﬂuencehaptics[31].
Moreover,somatosensoryreceptiveﬁeldsalreadyshrinkinapostnatalphasewhenonlyauditory,
butnovisualneuronsarepresent[279,277].Forthesecondaspect,behavioralandpsychophysical
studiesshowthatvisualperceptioncanevenbeinﬂuencedbyothermodalitiessuchashaptics[68]
oraudition[249,77,258].Moreimportantly,visionitselfcanimprove,respectivelysharpen,as
foundinthevisualsystemofyoungcats[279,277].
Altogethertheexperimentalandtheoreticalﬁndingswehavepresentedabovequestionthe
currentpictureofvision-guidedmapalignment[148].WallaceandStein[279]havepointedout
thatthedevelopmentofdiﬀerentmodalitiesstartsinparallelandintemporalcoincidencewiththe
appearanceofmultimodalintegration.Theyherebysuggestacommonmechanismdrivingboth
mapdevelopmentandmultimodalintegration.InChap.6,wewillsticktothisidea,namelythat
acombinationofallavailablesensoryinputintrinsicallyguidesitsalignmentbydevelopingthe
conceptofintegratedmultimodalteachingthataddressestheissuesmentionedabove.
Inthenextchapters,weturntomoresophisticatedsettingsthaninSec.2.4andgainnew
insightsintothecapabilities,characteristics,andlimitationsofthesessenses.

Esh¨ortdochjedernur,waserversteht.
JohannWolfgangvonGoethe

3.Auditoryprocessing:Exemplifyingoptimality

Inthischapterweapplyourmodelofoptimalmapformationtotheauditorysystem.That
is,signalweprodeducecessing,anaviz.,rchetypehandlingmodelechoandes.coBasedrresponondinourgmotempdel,oralwepreceptiveredicttheﬁeldsabilitforyofauditobothry
mathematicalandneuronalimplementationstosuppressechoesandextractsoriginalsignals
foinrm.vaInriousaddition,echo-richwecscenaomparios,reevenstate-of-the-ainthertabsencetechnicalofexactalgoinforithms,rmationbiologicalonthesolutionsspeciﬁcandechoour
atherchetcompaypemorisondelhintsderivedtofromlimitationstheofgeneralourgeneralframewoframewrk.orkOriginandatingpromisingfromthissitesparfoticrulafuturerwsetting,ork.

ductionIntro3.1inIntheChap.pres2toentachapterconcretewesettingapplyinthetheframewtempororkalofdomain.optimalInmapordertoformationdoso,wewetakepreviouslytheauderivditoryed
pathwayasareferenceforanarchetypemodelofsignalenhancementandechosuppressionthat
yieldoptimaltemporalreceptiveﬁelds.Togetstarted,weinitiallyreviewtheessentialfactsand
speciﬁcquestionsofinterestaccordingtothissetting.
Innaturalenvironments,auditorysignalsarefollowedbyechoes,orreﬂections,thatdegradethe
signal.Inthenon-biologicalﬁeld,thissettingiscalledreverberation.Notsurprisingly,biological
systemsprocessingacousticsignalshavedevelopedneuronalmechanismsofechosuppression
[20after,221the].originalHumans,signal,fordepexample,endingdoonnottheconsciouslysignalcpharactererceive[ec20ho].esWearrivingthereforelessexpthanectabdelaoutys20andms
suppression,inneuronalterms,inhibition,toshapethereceptiveﬁeldsofourmodel.Fora
ofquanthetitativreceptiveundeerﬁeldstanourding,mohodelwever,predicts?moreHospwdeciﬁcoesthisquestionsshapeneedconnecttobetoasked.biologicalWhatisrealitthey?shapCane
diﬀerentfunctionsbeattributedtodiﬀerentregionsofthereceptiveﬁeld?Canweidentifyuniversal
characteristicssharedbydiﬀerentreceptiveﬁelds?
Wehavealsochosenthecontextofauditorysignalprocessingbecausethisisasitewhere
hatecvehnicalalreadysolubeentions,realized.viz.,Thisalgorithmspromisesthataaddressfruitfulsomecomparisonoftheofabothevemoquedelsstionsandtoabiology.certainAlthoughextent
ﬁlters,artiﬁcialsimplealgorithmsandarerobustquitealgorithmssophisticatedmaywoutperemarkerformthatmore“[...]forsophisticatedacousticecsolutionsho[...]”cancellation[105].
So,thereareartiﬁcialalgorithmswithgoodperformancethatprovideenoughsimplicitytobe
settingcomparableandtoconthetrastarcourhetypmoedelmodelandatitshand.collectionIndoingofso,receptivweeapplyﬁeldsourtowhatpresenwtemoﬁnddelintoanatureneuronaland
technics.Finally,similaritiesanddiﬀerencesofthederivedarchetypemodelmaypointoutinwhich
wayalgorithmscanbeimprovedbymeansofbio-analogouscognitivearchitectures.

SignalEnhancementandEchoSuppression

3.2Archetypemodel
Inthissectionwebrieﬂyrecapitulatetheessentialstepsofourframeworkofoptimalmapformation
intheconcretesettingofauditorysignalprocessing.Thatis,wederiveanarchetypemodelof
signalenhancementandechosuppressionbasedonlyonthemathematicalprincipleofoptimality.

3.2.1Optimality:Anarchitecturalprinciple
Themodelderivationisbasedontheideaofsignalreconstructionbyoptimalreceptiveﬁeldsthat
inversethecorruptionprocessofechoesandnoise.Byderivingthenecessaryneuronalconnection
strengthsweformthebasisforasmoothintegrationofourarchetypemodelintotheexisting
knowledgeonauditoryprocessingsuchasrelatedphysiologicalandpsychophysicalresults.
Forthestartingpoint,themodelminimizestheexpectationvalueofthequadraticerror
E:=(st−sˆt)(st−sˆt)betweentheoriginalauditorysignal,s=(...,sµ,...),anditsoptimal
reconstructionsˆ=(...,sˆµ,...).ThereconstructedsignalsˆminimizesEandisthusoptimalin
theleast-squaresense.Thisisthedeﬁnitionofoptimalitywehavealreadyintroducedpreviously.
Theoptimalreconstructionsˆiscomputedbythetensorl.Itscoeﬃcientssuﬃcethesetof
equations(∂E/∂lµν=0).Theresultingreconstructionalgorithmcomputessˆonlybymeansofthe
sensoryresponser=(...,rµ,...)oftheauditorysystem,sˆt=ltτrτ.Thecoeﬃcientsltτofthe
matrixlaredescribinghow(inneuronalterms:withwhichconnectionstrength)avalueofthe
incomingsignalatthetimet−τismappedontotheoutputsignalattimet.Neuronallythiscan
berealizedthroughadelaylinewithdelaytimeτandasynapseofsuitablestrength.So,lcontains
thesetofoptimaltemporalreceptiveﬁeldsthatcompensateforthecorruptionoftheoriginalsignal
onse.respsensorythewithinThesensoryresponseitselfwillbe,inprinciple,aconvolutionoftheoriginalsignalandthe
transferfunctionhthatinthissettingwedenoteasechofunction,viz.,rt=htτsτ.Inthetechnical
ﬁeldtheechofunctionhismorecommonlyspeciﬁedas“roomimpulseresponse”(RIR).Itdepends
onthephysicalsurrounding,e.g.,the“room”.Inanalogy,thescenariobiologicallydenotedasecho
suppressionistechnicallyreferredtoasde-reverberation.
Asinthegeneralcase(seeChap.2),anyauditorysensorysystemalsohastocopewith
uncertaintiesnomatterwhethertheycomefrommeasurementerrors,varianceswithintheassumed
physicaltransmissionprocess,orsimplythroughthefactthatspaceandtimearecontinuous
quantitiesthatsensorsandespeciallyneuronscannotcontinuouslyrepresent.Artiﬁcialauditory
sensors,forinstance,typicallyaverageoveraspeciﬁcamountoftimeandgiverisetomeasurement
errors.Thetemporaldynamicsofneuronsareeveninherentlylimitedbycharacteristicssuchas
refractoriness.Inconsequence,theinput-outputrelationwithinanysystemwillbecorruptedby
errorsornoise.Wethusrewritethesimpleconvolutionrt=htτsτbyaddingthenoisetermς
accountingformeasurementandtransmissionfailurestooursensoryresponse.Inaddition,our
signalofinterestmay,andingeneral,willbedisturbedbycomplementarysignalswepoolandrefer
toasbackgroundnoisewemodelbyaddingarandomvariableαtothesignal.Wehencearriveat
rt=htτ(sτ+ατ)+ςt.(3.1)
Beforecalculatingthereceptiveﬁeldslweneedtodiscussiftheassumptionsregardingthe
expectationvalueswehavemadeinderivingthegeneralframework(2.16)arevalidinthepresent
contextofauditorysignalprocessing.Ofcourse,theinputsignalstisdeterministicandwewould
notexpectanyproblem.However,wedonotknowexantetheexactvaluesofst.Wecanovercome
thisproblembylookingat“biologicallyrelevant”signals.Suchsignalsbelongtoaclassofsignals
thatwedenoteas“typical”.Consequentlyaspeciﬁcsensorysignalisaconcreterealizationofa
classoftypicalsignals.Thatis,theinputsignalstisastochasticquantitywithadeﬁnedmean
µSandweuseanexpectationvaluetocomputeEinthesamewayasinthegeneralframework.

SignalEnhancementandEchoSuppression

Furthermore,wealsotakeallappearingcross-correlationstobezero,thatisnowthesignalvalue
atonespeciﬁcpointintimedoesnottellusanythingaboutthesignalvalueinthenexttimeframe.
Thesameholdstrueforbothtypesofnoise.Wetakethenoisetobeindependentatdiﬀerent
pointsintime,eachwithstandarddeviationσς/α.Pleasenotethattheallocativefunctionforthe
noiseneednottobeGaussian.We,asexpected,canthereforeusethesameassumptionsasincase
of(2.16).Thatis,onlytheautocorrelationsremainandwithδµνasKroneckerdeltaaregivenby
sµsν=µs2δµν,ςµςν=σς2δµν,αµαν=σα2δµν.(3.2)
Weﬁnallyobtainalinearequationforthecoeﬃcientsofthereceptiveﬁeldsldependingonly
onthetransferfunctionhandthedimensionlessparametersσ:=σς/µSandη:=σα/µS,
lµγhνδhγδ1+η2+σ2δνγ=hνµ(3.3)
wherehνδisthetransposeofthematrixhγδ.Theparametersσandηcorrespondtoinverse
signal-to-noiseratiosofσ–themeansignalstrengthtothevarianceofthedetectormessurement
errors–andofη–themeansignalstrengthtothevarianceoftheacousticnoise.Thereceptive
ﬁeldslnowmatchtheechofunctionhandallowcalculatingthereconstructionsˆoftheoriginal
auditorysignals.Sincewehaveneitherspeciﬁedtheoriginalsignalnorusedanyinformation
suchasinputcorrelations(3.2),ouralgorithmcanreconstructanyarbitrarysignal.Togetan
optimalreconstructionperformanceinthecaseofnoise,wesimplyneedtoadjustthetwomodel
parametersσandη.Ifwehaveaccesstothenoiselevelsσς/αandthetypicalvalueoftheoriginal
inputstrengthµS,thedeﬁnitionofσandηgivesusagoodestimateofthesevalues.Incaseprior
knowledgeofintrinsiccorrelationsoftheoriginalsignal,wecanincorporatethisintoµSthathence
transformsintoamatrixwithoﬀ-diadonalelementsandleavetheaboveminimalansatz.
Wehavenowderivedanarchetypemodelfromourframeworkofoptimalmapformationina
concretetemporalsetting.Inthenextsubsequentsectionwewillanalyzetheperformanceandthe
robustnessofthemodelbyapplyingittoexemplaryechofunctions.
3.2.2Receptiveﬁelds:Acomputationalbuildingblock
Byanalyzingthemathematicalmodelderivedintheprevioussectionwecanobservetheintrinsic
capabilitiesandcharacteristicsofourapproach.Wedealwiththreeexemplarybutexplicittemporal
transferfunctions,namely,theechofunctionshd;r;e;cf.Fig.3.1.Indoingso,wetakeadelta
functioninthetimedomainasinputsignal.Thisfunctioncorrespondstoaclick.Itappears
respslightlyonse.Thesmearedsimplestoutandenvirisfolloonmewntedbyfeaturinganecanhoecashowdeﬁnedouldbbyeathesingleechosolidfunctionwallininathefreesensoryspace.
ecHere,hothefunctionechowewouldassumebeaasingle,simple,wdiscreteeakenedrreﬂectionepetitionofaofclicktheandoriginlabelalthesignal.echo“d”Henceforforthe“discrete”.ﬁrst
Thesecondechofunction,“r”for“realistic”,mimicsatypicalroomimpulseresponse(RIR)where
wehavenotonebutmanywallsandwherethesignalisfollowedbyashortsilenceandmany
thesubsequenclick.Ftorthereﬂectionsthird[ec170ho].Wefunctiontakewetheseassumereﬂectionstheclosedtohavespaceatomaximbeevumenatmoreabout20restrictedmsafterand
hencedonotsupposeanygapbetweensignalandreﬂections.Herewetakeanexponentialdecayof
thereﬂectionsandthereforemarkitwith“e”.So,roughlyfollowingsomeexemplaryRIRs[105],
wecovertherangeofpossiblecomplexechofunctionsbythreesimpliﬁedscenarios,i.e.,explicitly
assumingthereﬂectingboundariesatτveryfar(case“d”),normal(case“r”),andclose(case“e”)
distance.Intensornotation(rt=htsτ),theyare
hdµν=exp−(µ−2ν)+exp−(µ−ν+2βd),(3.4)
22
2κ2κ
hrµν=exp−(µ−ν)2/2κ2+A(ν,µ),(3.5)

SignalEnhancementandEchoSuppression

Figure3.1:Threevariationsofechofunctionshwiththeoriginalsignalsandtheresultingsensory
respodiscretizednser;echosettingsfuncti“d”,ons“r”,hd,r,eand(bl“e”ack=from0,leftwhiteto=righ1t.)inThematrixuppernotationgraphs(asA−Cdeﬁned)depictby(3.4)the
–(3.6).Thelowergraphs(dD,r,−eF)showtheoriginalsignals(adeltafunction,ﬁlledline)and
resultingexemplaryecsensoryhorespfunctions,onsesvriz.,(soliddiscreteline)(left),inrealisticarbitraryunits.(middle),andThroughoutexponenthistialpaper(righwet)useecho,threeas
.2.3.2Sectioninexplained22heµν=expexp[−−((µµ−−νν))/κ/2]κififµµ−−νν<>00(3.6)
2wheredeﬁnesalltheconstancommontsarerealsynaptic(∈Ralph).aThefunction,functionµAand(ν,νµ):=denote(µ−theν+roβrws)expand−(µcolumns−ν+ofβr)h/,2γand
(µdela−yνb)etweindicatesensignaltheandrelativechoefordiscretizedconditiontime“d”anddiﬀerence.“r”.TheTheconstanparameterstsκβanddγandβdenoterdenotehowthethe
signalsandechos,respectively,getbroadened,andκisameasureforthedecayoftheexponential
“e”.onconditiintailWenowwanttoarriveatathoroughunderstandingofthereceptiveﬁeldsinvolvedinour
theoptimalthreemoechodel.Tofunctionsthisend,deﬁnedwebhayve(3.4)–calculated(3.6);cf.sensoryFig.3.1respandonsesFig.and3.2.Wreceptivehaeveﬁeldsalsoforeveacaluatedhof
theincludingrobustnessnoiseofandtheamo“mismdelbatcyh”comparingconditionwherereconstructedechofuandnctionoriginalandsignreceptivaleforvﬁeldsariousdonotconditionmatch;s,
cf.Figs.3.2and3.3.Importantly,wehavetestednotonlythecompleteoptimalreceptiveﬁeldsas
derivedintheprevioussectionbutalsoanabridgedversionthatdoesnotrequireanintegration
withwindoandwandwithoutthereforenoise,allowwsorkpropreal-timeerly;profordetailscessing.seeBothFig.3.2.abridgedandcompletereceptiveﬁelds,
andl.MoreoInver,doingwesohawveeusetestedthethewrongcollectionreceptivofeﬁeldsreceptivforeﬁeldsinreconstructingcaseoftheasignalmismatchandbetwcompareeenh
Fig.3.3reconstructionandshopwthaterformancethewithreceptivthee“matcﬁeldshing”(mainlylrcondition.)areAllableninetopcopeossiblewithcasesextremearevdepictedariationsin
–namelythewrongecho.Thesignalcanbereconstructedbymeansofreducingthenoiseand
eﬀectivelysuppressingtheecho;cf.Fig.3.3.Thisﬂexibilityisaprerequisiteforanybiologicalsystem
sincescenario.suchItastysurnstemoutcannotthataﬀordincreasingaspσeciﬁcmakesneuronalthewirindiﬀerengtforevreceptiverypeossibleﬁeldseclesshosensible,(suppression-)more
robust,andthusmakesthemperformbetterinthe“mismatch”condition,cf.Fig.3.3.

SignalEnhancementandEchoSuppression

toFigureright.3.2:GraphsReceptiv(Ae−C)ﬁeldslrepresenandtthereconstructedcompletesignalsreceptivsˆ;eforﬁeldscondild,r,etionas“d”,deﬁned“r”,inand(3.3)“e”(grafromy)andleft
theabridgedversions(black).Theversionsd,r,andealwaysappearasleft,middle,andright.The
areshoreconstructedwningraphssignals(Dobtaine−F)d.Thethroughbcompleteothabridgedreceptive(blacﬁeldsk)landleadtounabridgedhigherp(graeaky)receptivamplitudeseﬁeldsand
lessdepictedartifactsin(Gin−theI)withreconstructed10%noiseassignals.comAparednoisytosensorinputstillstrength.allowsAllforagraphsreliablehavebeenreconstructionexpressedas
inarbitraryunits.OriginalsignalandsensoryresponseareasinFig.3.1.

Toanalyzewhichpartofthereceptiveﬁeldscorrespondstowhicheﬀectinsignalreconstruction,
wehavesetselectedentriesinthereceptiveﬁeldstozeroandobservetheconsequences.Inresult,
e.g.forthe“realistic”receptiveﬁelds,theﬁrst,fastinhibitoryregionsharpensthecontourofthe
signalpeakandreducestheexponentiallydecayingtailoftheecho.Thesecond,slowinhibitory
regionreducesthesteepnessoftheechoonsetthatisimportantforauditoryobjectformation.

Tosummarizethepresentsection,wehavederivedandanalyzedasimple,yetgenericarchetypeof
modelforauditorystimulusreconstruction.Inthenextsectionwebrieﬂyreviewthestate-of-the-art
approachesintechnicalauditoryprocessingandrelatethemtothemodelathand.

3.3Technicalprospect
Havingderivedanddiscussedaveryfundamentalmodel,wenowrelateittomorerecentapproaches.
Fromthetechnicalpointofview,whatwehavederivedisasinglechannelde-reverberationalgorithm
thatusesonlyonefrequencychannel.De-reverberationiswellstudiedandanotyetcompletely
solvedproblemintechnicalaudioprocessing.
Ourapproachcoversawiderangeoftypicalde-reverberationalgorithms.Forexample,htτ=δtτ
correspondstoaconventionaldenoisingscenarioasdescribedby[66,290].Foraspeciﬁcecho
functionandσ=η=0themodelinverts,or,foraσ=0approximatelyinvertstheequivalentRIR
intheleast-squaresenseasdiscussedbyMiyoshietal.[195].Aspecialcaseoflinearﬁlteringisthe
parametersetofη=0andσ=1thatcorrespondstoasimpleWienerﬁlter[135,228,232,236];
forareviewonlinearﬁlteringsee[141].

SignalEnhancementandEchoSuppression

actualFigureec3.3:hoFlexibilitfunctionyhof(gratheymoarea)del.inAllarbitrarygraphsshounits.wtheIneachreconstructedcolumnasignaldiﬀerensˆ(blacteckholine)functionandtheh
isusedtocalculatethesensoryresponsewhereasthereceptiveﬁeldslarevariedineachrow.
Echomismatcfunctionhingtheandactualreceptivechoeﬁeldsfunctionmat(Bc,hC,onD,Fthe,G,Hdiagonal)can(A,leadE,toI).Evresonableenusingresultsifrecepσtivisechosenﬁelds
appropriately.Hereσ=5.Inthecasemismatchincreasedσleadstoresultsmuchbetterthanthe
initialσ=1whichcorrespondstotheWienerﬁlter[135,228,232,236].

Forabriefsynopsis,wehavetriedtosubdividethevarioustechnicalalgorithmsforde-
reverberationinthreegenerations.Theﬁrstgenerationconsistsofstraightforwardmethods,
thesecondcombinesseveralmethodsfromtheﬁrstgeneration,andthethirdgenerationtriesto
exploitthesamesignalcharacteristicsthatareexploitedbybiologicalsystems.Singlechannel
(usingonlyonefrequencychannel)de-reverberationoftheﬁrstgenerationcanbedistinguishedinto
threemajorgroupsofalgorithmsbasedonde-convolution,blindde-convolution,andsuppression.
Theﬁrstgroupassumesknownechofunctions(i.e.RIR)anddealswithapproximateandfast
methodsforinvertingit[105,195,206,215],asrealechofunctionsareusuallynotexactlyinvertible
[206].Thesecondgrouptriestoﬁndaﬁlterwhichaccountsforacertaincriterion,forinstancethe
meanpowerspectraldensityofspeechornoisereduction,andindirectlyreducesthereverberation
[66,73,90,105,273,290,291].Asinthepreviouscasethisﬁlterconvergestoanapproximation
oftheechofunctioninversion.Thethirdgroupestimatestheamountofechopowerspectral
density[22,105,173]andusessuppressionmethodstoperform,forinstance,minimummeansquare
estimationoftheoriginalsignal.Inthestateofartalgorithmsforechosuppression,theoneswerefer
toassecondgenerationechosuppression,diﬀerentmethodsarecombined[79,105,110,288].An
exampleforthethirdgenerationareattemptstorelyechosuppressioninanunknownenvironment
onthepitch[272,273].
Butstillthesealgorithmsdonotmeettherequirementsforartiﬁcialcognitiveauditoryprocessing
suchassoundsourcelocalizationorspeechrecognitioninadynamicreverberatingenvironment.In
contrast,oureverydayexperienceprovesneuronalalgorithmsthatallowsophisticatedauditory
processingexist.Therefore,inthenextsection,weshowhowthepresentedarchetypeforecho
suppressioncanbetransferredtoaneuronalsystem.Bycontrastingtheneuronalrealizationwith
actualneurobiologyweidentifysimilaritiesanddiﬀerences.Thisallowstopinpointpromisingsites
ofcognitiveprocessingintheauditorypathway,thatis,thesitesforfutureresearch.

SignalEnhancementandEchoSuppression

Figure3.4:Inﬂuenceofparameterσonthereceptiveﬁeldslfor“d”,“r”,and“e”formlefttoright.
Thereceptiveﬁeldsld,r,eforσ=1(black)andσ=5(gray)inarbitraryunits.Asσincreases,the
receptiveﬁeldsgetsmearedoutandloseﬁnestructure.Simultaneouslythereconstructiongains
generality,whichisadvantageousforreconstructingasignaldeformedbyanunknownechofunction,
seeFig.3.3.OriginalsignalandsensoryresponseareasinFig.3.1.

ectrosppBiological3.4Inthecourseofthisthesiswehavestatedthattheframeworkofoptimalmapformationcan
beimplementedtoaneuronalsettinginastraightforwardway.Inthefollowingweprovethis
statement.Ourpresentmodelofauditoryprocessingallowsaneasyone-to-onematchofthe
optimalreceptiveﬁeldswehavederivedabovetotheneuronalconnectionsincludingtheexplicit
valuesofdelayandsynapticstrength.Indoingso,wemapthereceptiveﬁelds(seeFig.3.4)toa
feedforwardnetworkasdepictedinFig.3.5.Suchaneuronalnetworkisthencapableofextracting
theoriginalsignalandsuppressingechoes.Theperformanceoftheneuronalrealizationisdepicted
inFig.3.6fordiﬀerentsituations,viz.,receptiveﬁelds,andvaryingσ.Justasexpected,thesignal
isextractedreliably.Thatis,weobtainaneuronalmodelthatsuppressestheechoandextractsa
signaljustasinthemathematicalmodel.Weherebydonotstatethatechosuppressionisdone
inbiologybymeansofasinglefeedforwardnetworkbutgiveanexistenceproofofourmodelin
are.hardwneuronalAcomparativelookontheshapeoftheoptimalreceptiveﬁeldsrevealsthattheresulting
synapticstrengthsofthedelaylinesfollowveryplausiblepatterns.Themostobviousfeatureisthe
emergenceoftwodistincttimescalesofsuppression.First,adirect,fastinhibitionwithaclear
maximum(e.g.atabout2mslastingca.5ms;cf.Fig.3.4B).Second,adelayedshallowinhibitory
region(inFig.3.4Bfromabout10-17ms)thatisthereforecalled“slowinhibition”.Thesetwo
ﬁndingsﬁtverywellintotoday’spictureofauditoryprocessing.Gapdetection,auditorycontrast
enhancement,andechosuppressionexploitpropertiesoftemporalreceptiveﬁelds,moreprecisely,
delayedinhibition[36,37,100,101,282].Asforechosuppression,itisunderstoodthatitisin
partmonauralandinpartbinauralasvonB´ek´esyandKoenigalreadystatedmorethan50years
[20].agoThemonauralpartofthesuppressioncorrespondstemporallyaswellasfunctionallytothefast
inhibitioninourmodel.Ithasamaximumatsimilartimes(about2ms[108]).Physiologicallyitis
locatedinthecochlearnucleusandhard-wiredneuronallywithoutsigniﬁcantvariability[283],just
asinourmodelwherethefastinhibitionisindependentoftheactualechoscenario.Inthemodel
thefastinhibitionsharpensthecontourofthepeakandreducestheexponentiallydecayingtailof
theechobutisnotsuﬃcientforcompleteechosuppression.Similarfunctioninghasbeenfoundin
andtheoreticallyexplainedforthecochlearnucleus[36].
Asforthebinauralpartofechosuppression,thingsaremorecomplicatedbynature.First,in
binauralprocessingmorenucleiareinvolved.Second,theyhavetodealnotonlywiththeinput
fromone,buttwoears.Third,theinvolvednucleireceivemanyeﬀerentﬁbers,thatis,top-down
controlplaysarelevantrole.Evenifitishardtodisentangletheindividualeﬀects,theapparent
resultsarethefollowing.First,binauralechosuppressionisslowerthanthemonauralone[283].
Second,itismoreﬂexible.Moreprecise,ithasbeenshownthatabinauralinhibitorymechanism
suppressesandchangestheconsciousperceptualprocessesofechoesdependingonthesituation

SignalEnhancementandEchoSuppression

Figure3.5:Neuronalsetting.AnoisyresponseR(t)incontaininganechoisusedtostimulateaset
ofPoissonneurons[115]A,theso-calleddetectorneurons.Theresultingspikespropagatethrough
delaylineswithdiﬀerentdelaysτ1,..,nandstimulatetheLeaky-Integrate-and-Fire-neuron(LIF)[88]
outputpopulationBthroughthecorrespondingsynapticconnectionstrengthsSc1,...,n.Synaptic
connectionstrengthsanddelaysaregivenbytheoptimalreceptiveﬁeldscontainedinl.Theoutput
Sˆ(t)outoftheneuronsBrepresentstheoriginalsignalwithsuppressedechoandreducednoise.

[221similar].timeComparedcoursetothatthetimestronglycoursedepofendstheonslothewisituation,nhibitioncf.foundFig.in3.4.nature,Foururthermore,modelitshoreduceswsa
thewherethesteepnessﬂexibilitoftheyecofhotheonset.binauralThisdepsuppressionendencyisupcononthetrolledbysituationtheﬁtstop-dothewneﬀebiologicalrents.counterpart
theInnaturalsummaryecho,asuppression.neuronalTheimplemenbinauraltationpofarttheofecarchohetypesuppressionmodelwthatorksisveryhighlywell.Itadaptableconnectscouldto
beasiteforfutureresearchtoidentifyandmathematicallydescribecognitiveprocessingsteps.

Discussion3.5Basedonthearchitecturalconceptoferrorminimizationandreceptiveﬁelds,wehaveappliedour
frameworkforoptimalmapformationtoaconcretesettingbydeducingaunifyingarchetypemodel
forreal-timeextractionofanacousticsignalfromacorruptedsensoryresponse.Wehavestudied
themodelperformanceinextractingtheoriginalsignalforthreeexemplaryechoscenarios.
Ontheonehand,wehaveshownlinkstocommontechniquesthatsolvetheinverseproblem
[242],applylinearﬁltering[141]suchase.g.theWienerﬁlter[135,228,232,236],pseudo-inverse
techniques,andtheMaximumLikelihoodEstimation[135,228,269].Ontheotherhand,wehave
implementedthearchetypemodelintoaneuronalsettingtocontrasttheoptimalandthebiological
approach.Initially,wehaveposedandansweredthequestionastowhichneuronaldelaysplay
aroleinechosuppression,whataretheirsynapticcouplingstrengths,andwhataretherelevant
timescales(seeFig.3.2and3.4).Wenotethattheresultingsynapticstrengthsofthedelay
linesfollowbiologicalpatternssuchasthetwotemporaldistinctinhibitoryregionsandhintto
cognitiveprocessingsuchastheCliftonEﬀect[46].Thatis,theauditorysystemadaptsquickly
tocharacteristics(echoesatabout∼10ms)ofaparticularsurrounding.Inmathematicalterms,
thesignalprocessingshowsplasticitybymeansofadaptingquicklytoaspeciﬁctransferfunction
onse).respimpulseom(roMoreover,auditoryobjectformation,bothspeechrecognitionandauditoryspacedevelopment,
areimpactedbythemechanismreferredtoasbinauralechosuppression.Thisisbecausetheslow
inhibitionreducesthesteepnessoftheonsetoftheneuronalresponsetotheechoandauditory
objectformationstronglyreliesontheonsetofacousticsignals[29,292].Theslowinhibition
isthusresponsiblefortheidentiﬁcationoftheechoasalocalizableobjectandaconsciousecho
perception.Inthislight,itmakessensethatitisrealizedbinaurallywithstrongtop-downcontrol.

SignalEnhancementandEchoSuppression

Figure3.6:Neuronalinputandoutputactivity;forsettings“d”,“r”,and“e”fromlefttoright.
AccordingtoFig.3.5,wefeedthesensoryresponseR(t)inwithanoiselevelof5%(ςt)toa
populationof140detectorneuronsA.Theresultingspiketrains(A−C)propagatethrough
thedelaylineswithdelaysτ1,..,nandstrengthsSc1,...,nandstimulateapopulationof140LIF
outputneuronsB.Thesetofdelaylinesrealizesthederivedoptimalreceptiveﬁeldscontainedin
l.TheupperrowdepictstheactivityofthedetectorneuronsAforthediﬀerentechofunctions
hd,r,e,thelowerrowthecorrespondingactivitySˆ(t)outoftheoutputneuronsBfortwosetsof
reconstructionparametersσ.Moreprecisely,thelowerrowdepictsthenormalizednumberofspikes
oftheoutputpopulationwithσ=10(D),5(E),and10(F)(graybars).Choosingtherightσ
enhancesthemodelperformanceascomparedtoσ=1(blackbars)thatcorrespondstotheWiener
ﬁlter.Obviously,neuronalcharacteristicssuchasthespontaneousrate,hereroughly15Hz,donot
degradetheperformancesincethesignalisextractedreliably.ThethresholdoftheLIFneurons
evenenhancesechosuppression.

Comingbacktothestate-of-the-artalgorithmsfortechnicalde-reverberationandtheirpotential
forimprovement,werealizethatthenaturalapproachismorepowerful.Today’sspeechrecognition
andauditoryobjecttrackingalgorithmscannotcopewithnoisecausedbyreverberationindynamic
scenes.Eventheidentiﬁcationofoneacousticsourceinthepresenceofothersisaproblemnot
entirelyovercomebytechnicalsystems.However,arobustauditoryrepresentationaspartofa
multi-modalrepresentationasfoundinanimals[147,261]issupposedtobeanecessaryingredient
forcognitiveapplicationssuchasrobotsandcognitiveworkbenches.Consideringtheeasewith
whichevennotfullydeveloppedbiologicalorganismslike,e.g.,ﬂedglingsmanagetoidentifyand
trackan,untilthenunknown,auditoryobjectsuchastheirmother’svoiceinanewenvironment
itisobviousthatthetwofoldviewonboththebiologicalandthemathematicalsideofthesame
problemdisemboguesintoafruitfulmergeofcultures.

Insummary,ourarchetypemodeluniformlytiesupwiththeknownphysiological,behavioral
mapﬁndings,formationtechnicalisaaspfeasibleectstoandoltomethoprodsvideofaechocommonconsuppression.vergingThpaterspis,ectivoureonframewsensoryorkofprocesoptimalsing
enablingthemultidisciplinaryapproach–bionics.

erceptionPLateral-line4.

Wirk¨onnendieWeltnichterkennen,wiesieist,
sondernnur,wiesief¨urunsist.
erretrandersNoorT

Thelateral-linesystemisauniquemechano-sensoryfacilityofaquaticanimalsthatenables
themnotonlytolocalizeprey,predators,obstacles,andconspeciﬁcsbutalsotorecognize
hydrodynamicobjects.Inthischapter,wepresentananalyticapproachthatshowstheeﬀectof
submergedmovingobject(SMO)shapetolateral-lineperception.Basedontheseresultswe
deriveanexplicitmodelexplaininghowaquaticanimalscandistinguishdiﬀerentlyshapedSMOs.
Ourmodeloriginatesfromthehydrodynamicmultipoleexpansionandusestheunambiguousset
ofmultipolecomponentstoidentifythecorrespondingobject.Furthermore,weshowthatwithin
thenaturalrangeofoneﬁshlengththevelocityﬁeldcontainsfarmoreinformationthanthat
duetoasimpledipole.Finally,theresultswepresentareeasytointerpretbothneuronallyand
technically,andagreeswellwithavailableneuronal,physiological,andbehavioraldataonthe
system.lateral-line

ductionIntro4.1AllﬁshandsomeaquaticamphibianssuchastheclawedfrogXenopuspossessauniquesensory
facility,thelateral-linesystem.Thissystemiscomposedofmechanosensoryunitscalledneuromasts
locatedonthetrunkoftheanimal.Theneuromastsconsistofsmallcupulae,gelatinousﬂags
protrudingintothewater,whicharesensitivetothelocalwatervelocity[21].Theabovementioned
animalsusetheirlateral-linesystemtolocalizepredators,prey,obstacles,orconspeciﬁcs.The
laterallineissoeﬀectivethatitenablesevenblindﬁshtonavigateeﬃcientlythroughtheir
environmentbymeasuringandanalyzingthepressureandvelocityﬁeldofthesurroundingwater
[52,111].Moreover,itissupposedthataquaticanimalsnotonlylocalizebutalsoextractand
recognizefeaturesfromthehydrodynamicstructureofthevelocityﬁeldtodeterminesize,speed,
andpresumablyshapeoftheobjectgeneratingit[39,52].Thiscanbedoneevenindirectly,for
instance,bymeansofextractinginformationfromthevortexeswithintheobject’swake[52].

Thequestionwenowpose,andanswer,iswhetherandhowapassivedetectionsystemsuchas
thelaterallinecanbothlocalizeamovingobjectanddetermineitsshapeinanincompressibleﬂuid
suchaswater,orairatlowvelocities.Moststudies,bothexperimentalandtheoretical[54,75],
usedavibratingortranslatingsphereasstimulus.Duetothespecialsymmetryofthesphere,the
resultingvelocityﬁeldisexactlythatofadipole–nomatterwhethervibratingortranslating[171].
Ofcourse,notallobjectsinnaturearespheres;cf.Fig.4.1.Onlyforvibratingbodiescanone
identify[107,142]theinﬂuenceoftheirshapeontheﬂowﬁeld,showingareasonabledominanceof
thedipole.Theliteraturelacks,however,ageneralexplanationinamoreglobalcontext.Herewe
showandquantifyhowmuchinformationisavailableinthevelocityﬁeld,howonecanextractit
andtowhatextentindependenceuponthedistance.

ineLateral-l4.erceptionP

Figure4.1:Submergedmovingobjects(SMOs)appearinwidelyvaryingshapes.Furthermore,
aquaticstimulimaybutneednotmoveatall;forexample,vortexstructuresthataregeneratedin
thewakeofaswimmingﬁshorattheendoftheﬁns[52].Thesestimuliimprintinformationonthe
ﬂowlateral-lineﬁeldthatorganscanisbemoreaddiﬁedoutbdueytothequanlateral-linetitiessuchorgans.astheInSMO’saddition,ptositheion,inputmotion,measuredandatshapthee.
Thebodyofthedetectingﬁshalsoinﬂuencestheﬂowﬁeld,forinstance,bymeansofdamping
(FromNOAA’sHistoricFisheriesCollection;IDs:ﬁsh3078andﬁsh3039)

Moreinparticular,weﬁrstderiveanalyticalequationsshowingthatshapeinformationplays
anessentialroleaﬀectingthevelocityproﬁleatthelaterallinewhichisthenaturalstimulusto
inthetermsmecofahanosensorymathematicalsystem.Second,descriptionweinthreedemonstratedimensionshowhisydropossibledynamicbyobjectmeanscofamharacterizationultipole
apreexpansiondator,can(ME).readInoutaddition,thevweloecitshoywﬁeldhowandanaquaticreconstructanimal,theshapcalledeofasdetectingtimulus,animalviz.,(DaA),subsmuchergedas
movingobject(SMO),suchasprey.Finally,wegivelinesofevidencethattherearebiologically
therelevanrangetuplimitstoofwhichstransmittinghapeandinformationextractingisimportaninformationt.The–forlatterboth,hasthenotbeencapabilityexploredtoloupcalizetonoandw.

4.2Analyticdescriptionofshape
Asexplainedabove,disturbancesofthesurroundingwaterformthenaturalinputtothissensory
system.Theinputmay,thoughneednot,begeneratedbysubmergedmovingobjects(SMOs)and
dependsontheirposition,motion,andshape.Thebodyofthedetectingﬁshalsoinﬂuencesthe
ﬂowInﬁeldthis,cforhapterinstance,wefobycusonmeansandofisolatedampingthewaterinﬂuenceofdisturbances,theSMOviz.,shapthee.signal;Tocf.outlineFig.the4.1.basic
poideawoferfulthetwpresento-dimensionalsection–anmethodsanalyticalbasedonthedescriptioncomplexoftheforminﬂuenceulationofofstimpotenulustialshapﬂoew–devweeloprecalled
forairshipdesign[270].Potentialﬂowisavalidhydrodynamicdescriptionofthelateral-line
perceptionalthoughitneglectseﬀectsofviscosity[95,111].Acomplexpotentialﬂowdescription
ofthree-dimensionalhydrodynamicsettings–includingevenrotational-symmetricbodies–is
ellipticallyanalyticallyshapnotepdossibleproﬁles(seewithinApptheendix(x,yB).)Wplane.eInthereforedoingso,reducewethegainobjectsanalytictoalaccesscircularlytoandthe
essentialeﬀectsofobjectshapeontheﬂowﬁeld.

4.Lateral-linePerception

a=Figure10−43.2:,2.5Flo,3w.5;aroundtheradiusthreer=diﬀeren4,andtlyV0shap=ed1.Weproﬁles.indicateFromthelefttoproﬁlerighbyt:grashapy.eOneparametercanseeho(4.2)w
theproﬁlebecomesellipticandthestreamlinescorrespondingtotheinputtothelaterallineadapt
accordinglytoobjectshape.

rmulationfoMathematical4.2.1ThecomplexvelocitypotentialF(z)candhencethecomplexvelocityw(z)c:=dF(z)c/dzwith
x+iy=:z∈C,aroundacircularproﬁle[218]isgivenby
F(z)c:=V0z+r2(4.1)
zthewhereJoukV0owskidenotesthetransformationmovingspζ(eedz):=andzr+athe2/zradiuswithofshaptheecircularparameterproﬁle.a∈InR[the218],nexti.e.,step,wconformaleuse
mapping,toobtaintheﬂowﬁeldwe(z)aroundanellipticallyshapedproﬁle.Herethesuperscripts
ctheandﬂoewreferﬁeldwtoeincirculartheζand-planebutelliptical,intherespectivoriginalely.planeInconoftrtheasttosphere,thetheusualzwa-plane.y,weThwillatis,notwestudycan
identifythecoordinatesandthuscomparetheﬂowofthecircularandellipticalproﬁle,
edFdzr2a2−1
w(z)=dzdζ=V01−z21−z2.(4.2)
wc(z)ddζz
Inthiswaywehavederivedthecomplexvelocitywe(z)oftheﬂowaroundanarbitraryelliptic
eandshape.theFuvelocitrthermore,yﬁeldwveecan=(ve,computeve)byallinmeansterestingof(4.2)quanandtitiesz=xsuc+hiyas.theAllvelothesecityptermsotenaretialΦlisted(x,yin)
Table4.1.Thetransformationxyfromtheﬂowaroundcircularproﬁlestotheﬂowaroundelliptical
onescanbedescribedbyonlyoneparametera∈R.Asa→0,allellipticalquantities(seeTable
4.1)convergetothecorrespondingquantitiesofthecircularﬂow.
Wesummarizethesethoughtsbystatingthatthecircularﬂowisonlyaspecialcaseofthe
moregeneralellipticﬂow(4.2).Asaconsequence,weobviouslycannotexpectaprioristatements
derivedfromcircularscenarios,i.e.,thespecialcase,toholdgenerally–ifatall.

eshapofEﬀects4.2.2Havingderivedageneralexpression(4.2)fortheﬂowaroundanellipticallyshapedproﬁle,wenow
exhibitfundamentalmodiﬁcationsofthevelocityproﬁleasitismeasuredbythelaterallinedue
tostimulusshape.Inthepresentliterature,onedistinguishestwoclassesoftheoreticalmodels
thatexplainthedetectioncapabilityofthelateral-linesystem[54,75,95,251].Modelsoftheﬁrst
thecategoryﬁsh’sboexplaindy.Thethepseerfcondormanceclassofofmothedelslateral-lineexploitschsystemaracbteriysticusingpointhetssucwholehasvelocitextremyaproﬁleandzerosalong

erceptionPineLateral-l4.

Figure4.3:TherelativequadraticerrorΔofthevelocity-proﬁleextremaatthelaterallinexEin
dependenceuponthedistanced/randshapeparametera/rnormalizedbytheradiusr.Thegraph
showsΔ:=[xE(0)−xE(a)]2/xE(0)2ford/r=0.5,1,1.5.Fordistancesdthatarecomparable
totheSMOsize(radiusr),viz.,inmathematicalterms(d≈r⇔d/r≈1),thepositionsofthe
extremaxEclearlychangewiththeshapeparametera.Presentmodelsignoringshapeeﬀects
calculatingthedistancedfromalinearrelationd∼xEthereforeoverestimatetherealdistance
beyondd≈r.

ofthevelocityproﬁlealongthelaterallinesincethesepointsaresupposedtobeeasilyaccessible
.neuronallyWeﬁrstanalyzethelattercategory.Knowingtheorientation,i.e,thedirectionofsensitivity
g=realizedg(x,byy)andmeansposofivtiong(rr)==r(dΦx,/ydg).ofIntheotherwlateral-lineords,weorgans,canevcalculateerythelateral-linewatervelogeometrycityincanevberye
directiongandatanypointroftheﬂowﬁeld.Here,foreaseofcomparisonwithcommonscenarios
inthethedetectingliterature,animal.weInstudythisansetup,SMOcenthevteredelocitaty(0,proﬁle0)ispassingidenticalalongwiththevx-axisandatyadenotesdistanceyexactlyto
xtheWedistancestartbdybresetwcalineengtheourcentersexpressionoftheforvSMOxbyandmeanstheoftrunknormalizinglateralline.allquantitiesbytheSMO
size,i.e.,theradius.Thisresults,forinstanceina→(a/r)r=ar.Thenormalizationmakesv
explicitlyindependentofr.Computingthex-valuesxEoftheextrema(skippingxE=0)bymeansx
ofcomputing∂vx/∂x|x=xE=!0givesxE(a)=±a2+dd±2√a2+d21/2.Obviously,the
extremadependontheshape;inmathematicalterms,ontheshapeparametera∈[0,1].Onlyin
thecaseofacircularproﬁle(a→0)dotheextremaencodethepuredistancexE(0)∼d.Common
tomothedelszerosexplaind∼the(xloE1−calizationxE2)[p54,75,erformance95].oftheWithoutlateral-linetakingthesystemshapbeyintomeansaccounofat,linearthepresenrelationt
modelsthereforeoverestimatetherealdistance.
detectingAtthesamelateral-linetimesystemtheaseﬀectwellofasshaptheeSMOalsobdepodyendslengthonthethatdistancedirectlybetrelatesweentotheSMOandparameterthe
rangeradiusrdeﬁnedintheastheequations.distanceAsupwetocantheseeSMOinbFig.ody4.3,lengththeandinﬂuvenceanishesofforshapelargerdominatesdistancesthednearb1.y
Nextweturntotheotherclassofmodelsthatnotonlyexploitparticularpointsbutthewhole
pvelootencitytialﬁeldofatellipttheicallateralproﬁlesline.Fe(Tzo)inanalyzeaTaylortheseriesinﬂuencearounofdshapa=e,0,weexpandthecomplexvelocity
Fe(z)=V0z+rz+rz3V0−Vz0a2+O[a]4.(4.3)
22
Fc(z)
Throughthesubstitutionc:=r2/a−awecantransform(4.3)intoapowerseriesinz,
Fe(z)=V0z+V0ca+a3+a5+O[a]7.(4.4)
zzz

4.Lateral-linePerception

Table4.1:Hydrodynamicetoolbox:Comparison2oftheﬂowaroundcircular(denotedbyc)and
theellipticalﬂowaround(denotedanbyellip)ticalproﬁles.proﬁlebDeﬁningymeancs:=ofr/aonly−a,oneweshapcanederiveparametersimplea.equationsdescribing
Fc(z)=V0z+rz
Complexvelocityp2otential
Fe(z)=V0z+2c[ln(z+a)−ln(z−a)]
Hydrodynamicpotential
22Φc(x,y)=V0x+x2r+2xy2
Φe(x,y)=V0x+4cln((xx−+aa))2++yy2
Velocityoftheﬂowinx-direction
vxc(x,y)=V01+x2r+2y2−(x22r+2yx22)2
vxe(x,y)=V01+2[(xc+(xa)+2a+)y2]−2[(xc−(xa)−2a+)y2]
Velocityoftheﬂowiny-direction
vyc(x,y)=−V0(x22r+2yy2x)2
22vye(x,y)=−V0a4−2a2(2(x2r−−y2a))+(yxx2+y2)2
Equation(4.4)showsmoreclearlythan(4.3)thattheshapeparameteraisthenaturalexpansion
parameterforthecomplexmultipoleseriescharacterizingshape.Wewillexploitthederivedﬁnding
thatmultipolescanbeusedtodeﬁneandcharacterizehydrodynamicobjectshapelateron.
Thediﬀerencebetweentheﬂoworiginatingfromacircularandanellipticallyshapedproﬁle,
andthustheimpactofshape,canbeexpressedtoﬁrstorderby
22ΔF(z)=[Fe(z)−Fc(z)]=−arV0+O[ar]3(4.5)
zwherea=a/risagainnormalizedandrangesbetween0and1.Itiseasytoseethatthecomplex
velocitydiﬀerenceΔw(z)=dΔF(z)/dzandthusthemeasuredvelocitydiﬀerencescalesas(r/z)2.
ThebodylengthoftheSMO(≈r)thereforedeﬁnesanaturallengthscaleuptowhichtheeﬀectof
shapecontributessubstantiallytotheﬂowﬁeld.Withinthisrangethedipoleapproximationis
onlyavaliddescriptionforcircularshapes.
Insummary,shapeshiftsthepositionofcharacteristicpointsofthevelocityproﬁleandalso
modiﬁesitasawhole.BotheﬀectsdependontherelationbetweendistanceandSMOsize.Since
underpotentialﬂowconditionwehavep∼vv[95],theSMOshapeaﬀectspaccordinglyand
thusalsothecanallateral-lineprocessing.Moreoverwehavegivenananalyticallineofevidence
thatthenaturalrangeuptowhichinformationaboutshapeistransmittedistheSMObodylength.
Havingdiscussedreallysimplebutneverthelessnon-trivialshapes,wenowwanttogivean
outlookonpossibleextensionsofourmodel(4.2).Itisstraightfrowardtoadapttheequations
inTable4.1tomoreﬁsh-likeshapes.Substitutingz→(z+b)withb∈RintheJoukowski
transformationleadstomorecomplicatedbutricherexpressionswheretheadditionalshape
parameterbinreturnallowsformorerealisticshapes.Wehavedepictedsuchshapesandresulting
velocityproﬁlesinFig4.4.
Inthenextsection,however,weturntothree-dimensionalshapes,theirinﬂuenceonlateral-line
perception,and,moreimportantly,howthelaterallinecanrecognizesuchdiﬀerentlyshapedSMOs.

erceptionPineLateral-l4.

Figure4.4:Eﬀectofthesecondshapeparameterb.Thegraphsdepicttheﬂowﬁeldaroundthree
diﬀerentproﬁleswithvaryingthesecondshapeparameterb=0,−0.1,−0.3(fromlefttoright).
TheradiusoftheSMOisr=1andtheshapeparametera=0.5from(4.2);bothrandaare
constant.Theproﬁleitselfisindicatedbythegrayarea.Duetovariationofbtheproﬁlebecomes
moredrop-shapedandthestreamlinesadaptaccordingly.

recognitionobjectdynamicHydro4.3mInultipthisolespart,iswecapablewanttotopicdescribkeupthetheinﬂuenceﬁndingsooffSMO(4.4),shapnamee,lyandthatderivaeseriesbothofahydromathematicaldynamic
deﬁnitionanddescriptionofhydrodynamicobjects.Inaddition,wederiveanoptimalobject
inthreerecognitioncoordinates.algorithmInthatdoingcanso,lowcalizeeareandabletoreconstructpredictptheositionandcapabilitiesshapofeoftheSMOlateral-linesimultaneouslysystem.

4.3.1Deﬁnitionofhydrodynamicobjects
Aswediscussedintheprevioussection,avelocityﬁeldrepresentsanadequateandnaturalstimulus
tothelateral-linesystemandcanbedescribedbyamultipoleexpansion(4.4).Now,weleavethe
two-dimensionalspaceandfocusonthreedimensions.Becausetherelevantﬂuiddynamicsarewell
realdescribveloedcitbyyptheotentialEulerv(r)=equation−[Φ(95r])w[e171tak]efortheathree-startingdpimeoinnst.ionalUsingvelothecityrealﬁeldsphericalderivedfromharmonicsthe
YlRmandthesphericalmultipolemomentsq=(...,qlm,...),wecanexpandthevelocitypotentialΦ,
llR∞∞Φ=qlmΦlm=qlm1Ylml(θ+1,ϕ)(4.6)
l=1m=−ll=1m=−l2l+1r
describForedthesakthrougheofcosimplicitordinatesyweς∈fo[0cus,α]onandηrotationally∈[0,2π],symmetricalbodieswhosesurfaceScanbe
−1/3
ςπ√2γsin(α/2)sin(ς)cos(η)
S(ς,η)=Ncosα3×√2γsin(−1/α/32)sin(ς)sin(η)(4.7)
γ2/3cos(ς)
Thethatvsurfaceortexstructuparametersresαcan∈be(0,πdesc/2]ribandedbγy∈(0means,∞)ofadeterminemultipoletheshapexpanesofionthe(ME)surface.aswellWe[171note].
BecauseSMOssuchasﬁshleavecharacteristicvorticesintheirwake,theﬁndingswederivein
thisandthefollowingsectionsarealsoapplicableforidentifyingSMOsindirectlybymeansof
characteristicvorticesintheirwake.
BecauseoftheEulerequation,onlytheNeumannboundaryconditionv∙nS=0isrealizable
wherenSdenotesanormalvectortothesurfaceS.Intermsofsphericalcoordinateswecalculate

4.Lateral-linePerception

Figure4.5:Diﬀerentshapeparametersresultindistinguishablesetsofmultipolecomponents.We
depictanapproximationforonlythreemultipolemomentsl≤k=3in(4.8).TheSMOsurface
parameterαincreasesalongtheverticalaxisandtheSMOsurfaceparameterγalongthehorizontal
one.Wehavecalculatedthesetofmultipolesqˆthrougharasterof30,000randomlydistributed
positionsoneachmovingobject’ssurfaceandusedobjectswithlengthsofabout5cmanda
matchingmovingspeedv∞=0.01m/s.Eachvolumeisnormalizedtothevolumeoftheunitsphere.
Itisthereforefairtosaythatthedipoleq10staysalmostconstantwhilethequadrupoleq20varies
accordingtoαandtheoctupoleq30accordingtoγ.Therelativeerror(k):=|Tqˆ−w|/|w|on
theSMOsurfaceSascalculatedforrotationallyinvariantellipticbodiesgives(k)<0.4fork=3
andγ<2.Itconvergesslowlytoabout0.15fork→∞.Forγ2amultipoleexpansioncannot
describethecorrectvelocityﬁeld.This,however,isnottoorestrictive.Forexample,theﬁshin
Fig.4.1canbedescribedbyγ≈1(leftﬁsh)andγ≈2(rightﬁsh).

thecomponentsofv(r)=−Φ(r)=(vr,vθ,vϕ)atrthroughcoeﬃcientsarlm,etc.,sothat
∞l∞l∞l
vr=l=1m=−lqlmarlm,vθ=l=1m=−lqlmaθlm,andvϕ=l=1m=−lqlmaϕlm.(4.8)
Thegeometry-depequationsending(4.8)cancobeeﬃcienwrittentsarlinm.Tmatrixoexplformicitlyv=AcalculateqwiththethesinglematrixmAultipconoletainingmomenthetsqlonlym,
weuniformhavesptoeedspvecifyofthethebSMOoundaryintheconditionstationarywithrespframeectoftonSreference.and,forThethebsakoundaryeofsimplicitconditiony,thenthe
∞readsv(rS)∙nS=nS∙v∞=nSTA(rS)qwithrS∈S.(4.9)
theThevelocitinﬂuenceyﬁelofdtheonlymupultiptoaolesincertain(4.8)numbdecreaseserkandwithsetqlmincreasing=0lfor;lsee>k(4.6)..WeHencearethwuseexpanddealing
withanapproximatedsetofmultipoles,denotedbyqˆ.Toﬁndanappropriateapproximation
wassoetakciatedesevsurferalacpeositionsnormals1n≤i,i≤thenmatrixrandomlyAi,anddistributedwriteitonlineanbySMOlineinsurfacetotheandtransfercalculatematrixthe

ineLateral-l4.erceptionP

T:=(...,niTAi,...).Theindexiofthevelocitycomponentswi:=v∞∙niofthevectorw
spcorrespeedvondsoftotheniSMOandleadsthereforetolabincreasingelsthevsamealuespofosition.thecorrespHere,weondingnotemthatultipanolemomenincreasingts.moInvingthe
∞nextsection,thiswillbecomeimportantinthecontextofthedetectororganssignal-to-noiseratio.
TocalculatethemultipoleswesimplysolvethelinearequationTqˆ=w.Themultipolemoments
followfromqˆ=(TTT)−1TTw[228].Thesolutionapproximatesthemultipolecoeﬃcientsuptoa
givenkoptimallyinthesenseofminimalquadraticerror.Introducingacharacteristiclengthscale
λ,saythebody-lengthoftheSMO,intoequation(4.6)weget
v(r)=qlm(λ/r)l+2flm(θ,ϕ)(4.10)
,mlwhereflm(θ,ϕ)arefunctionsdependingonlyontheangularcoordinatesθandϕandcanbe
calculatedbymeansof(4.6).Indoingso,weobtaindimensionlessmultipolemoments.Onthe
onehandwecancomparethemindependentlyoftheSMOsize.Ontheotherhandweseefrom
v(r)=l,m(λ/r)l+2qlmflm(θ,ϕ)howtheinﬂuenceofeachmultipolevariesindependenceupon
thetellsussizeλthatofthetheshapSMOeofandthetheSMOisdistanceonlytoimptheDortanAtlabiftheledebsizeyλtheandradialthecodistanceordinaterarer.ofThistheresultsame
willnotmagnitude.transmitItisane.g.yobshapeviousthatinformationplankton,throughwhicthehisﬂowtenﬁeldtoh–andundredinthistimescasesmallershapethantheinformationDA,
isnotneededeither.Thisresultisinperfectcorrespondencywithwhatwehavederivedpreviously
startingfromdiﬀerentequationsinthecomplextwo-dimensionalspace;cf.Fig.4.3.
However,asstatedbefore,inschooling,mateﬁnding,orpredator-preybehaviorobjectinforma-
tioncanmakeadiﬀerence,andasshowninFig.4.5,diﬀerentlyshapedobjectsarerepresentedby
diﬀerentsetsofmultipolemoments.Consequently,multipolesareabletoquantitativelydeﬁneand
describehydrodynamicobjectsaswehavedemonstratedfortheﬁrsttime[251]–asfarasweknow.
4.3.2Objectrecognitionperformance
How,then,canaquaticanimalssuchasﬁshreconstructtheabovemultipolemomentsfromwater
velocitymeasuredthroughtheirlateral-linesystem?Ofcoursetheanimalhastofulﬁllthistask
undertheinﬂuenceofomnipresentnoiseandthelimitationsofitsneuronalsystem.
Ourreconstructionmodelforthree-dimensionalshaperecognitionisbasedonamaximum-
likelihoodestimator[276]andtheframeworkofoptimalmapformationthatwehaveintroduced
inChap.2.Thatis,wearelookingforthemultipolesqwiththehighestprobabilitygiventhe
measuredvelocitieswontheDAbody.Sotospeak,thisestimatormaximizestheconditional
probabilityp(q,r0|w)fortheSMOcenterofmassr0,whichhasalsotobedeterminedbymeansof
theinformationavailablethroughtheDAsuperﬁcialneuromasts,thewatervelocities.Thevelocity
sizewiatorganiisgivenby
wi=T(r0,ri)q+ni.(4.11)
WenotethatthetransfermatrixT(r0,ri)dependsontheSMOpositionr0andtheposition
riofthelaterallineorgani(DA).Asintroducedintheﬁrstchapter,thetransfermatrixmaps
thesignalontothedetectors.InthiscaseT(r0,ri)projectsthemultipolesqlmontothevelocity
componentswimeasuredatlateral-lineorgani.NoiseismodeledbyaddingindependentGaussian
randomvariablesniwithmean0andstandarddeviationσntothewi(4.11).Allpositionsr0areof
equalprobability.WethereforetakeaGaussianprobabilitydistributionp(q,r0)∼exp(−q2/2σq2).
InsteadofmaximizingthecombinedprobabilitybasedonBayeslaw,wemaximizeitslogarithm
L(q,r0):=ln[p(q,r0|w)].Deﬁningthedimensionlessparameterσ:=σn/σq,wejusthaveto
maximizeL(q,r0)=−[w−T(r0,ri)q]2+σ2q2(4.12)


4.Lateral-linePerception

Figure4.6:Neuronalimplementationof(4.14)isstraightforward.Inaﬁrststep(A)→(B)the
vmeloultipcitiesolewicompattheonentsqˆlmneuromasts(r0)((BA)).areThiscalculatedcorrespforondstodiﬀerenanettwporkositionsofr0synapticfromtheconnectionsmeasuredbetwweenater
thelcomputedateralfromlineandtheenthetriescenoftralthenervrighoust-handsystem.sideTheof(4.14)strengthorofcanthebelearnedindividualneuronallyconnections[74].canInbae
secondconnections;step(Bcf.)→(4.12).(C),LThe(q,mrax0)imisumL(calculatedqˆ,rˆ0)fromindicatestheqˆrthe0,whiccorrecthpcanositionalsobofethedoneSMObyandfeedforwselectsard
theoptimalqˆr0fromstep(C)→(B);cf.(4.13).

thewithlikrespelihoectodtoL(theqˆ,rˆm0)ultipattheole-momencorrectlytvectorestimatedqandpr0.ositionTherˆ0isnecessaryconditionforamaximumof
∂L(q,rˆ0)|q=qˆ=TTT+σ2q−TTw|q=qˆ=0,(4.13)
q∂whichleadstoalinearsystemofequationssolvablebymeansofthepseudo-inversetechnique[228],
qˆ=TTT+σ2−1TTw.(4.14)
AWeneuronestimatealtheimplemensetofmtationultipofolesthqˆesegivenonlycalculationstheismeasuredstraighvtforwelocitiesardwandatcanthebedonelateral-lineeasilyorbgans.y
neuronalhardware;seeFig.4.6.Thusthepositionandtheappropriatemultipolemomentscanbe
.neuronallycalculatedTotestwhetheritispossibleforaquaticanimalstodetermineSMOpositionandshapeusing
theabovemethod,wehaveappliednoisetotheinputsignalwi.Wehaveusedσn=10−4m/sas
standarddeviationofthenoise,whichcorrespondstothevelocitythresholdofXenopus’lateral
lineorgan[21].Furthermore,wedeﬁneasignal-to-noiseratio,SNR:=v∞/σn.Plausiblemaxima
ofthemovingspeedofSMOsrangebetweenonebody-lengthpersecondandzero.Tobefair,in
theexamplesintheﬁgureswehaveusedavelocityaslowasv=1cm/scorrespondingtoonly
1/5to1/10ofthetypicalmaximummovingspeed.Thatistosa∞y,wehavedealtwithaworse-case
scenariosinceincaseofhighermovingspeedv∞thereconstructionperformanceisevenbetter
thanpresented.Anotheraspectregardinglateral-lineperformanceisthenumberofdetectororgans.
Withinabiologicalrangetheinﬂuenceoftheexactnumberoflateral-lineorgansismarginal.We
butshowstainysFig.almos4.9tthatconstanthetfordiﬀreconstructionerentnasumwbellersasoftheloneuromasts.calizationOferrorcourse,increasestaking,withfortheinstance,distanceten
timesmoreneuromastscancompensateahigherSNRvalue–butonlywithinmarginalranges.In
summaryAnother,theissueSNRis,thedominatesmoremtheultippoleswerformanceeuseasthemorecomparedptoositionsthearenumofbherighofprobabilitneuromasts.ybecause
onethehasshapemoreestimateparametersimprovtoes.ﬁtWtheecancmeasuredhoosewian’s.Thusappropriatetheσestimatedforapcertainositionktogetscompambiguousensatebutthis

ineLateral-l4.erceptionP

Figure4.7:Thelateralline(A,grayline)iscenteredat(0,0).Thedistancebetweentheobject
(SMO,grayarea)centerofmassr0andthelateralline(DA)isjustbelowthelengthofthelateral
line.Usingthedipoleterm(k=1)only,wecaneasilyreconstructthepositionrˆ0(A,cloudof
mblacultipkpoleointsmomenaroundts(rk0=),3a)asresultaneﬃcienconsistenttwiminimalththatmoofdel,Fourranoschmethoetdal.[allo74ws].Ifaweproptakereonlyestimatethreeof
bothposition(A,smallopencircles)andform,cf.Fig.4.8–eventhoughmorepositionsarenow
likelytooccur.Forbothcases,40reconstructionshavebeendepictedwithσn=10−4m/smoderate
SmapNRof:=pv∞osition/σn=100estimation.andIt500showsneuromasts.noamBbiguity;depictscf.logFig.[L4.(qˆ,6C.r0)]correspondingtotheneuronal

eﬀectandenableafaithfullocalization;fordetailsofhowtotuneσconsiderChap.2.InFig.4.7
weshowtheabilityofourmethodtoestimatetheSMOpositionrˆ0.
Followingtheansatzoftheneuronalmodel(Fig.4.6)wehavecalculatedthemultipolemoments
attheestimatedposition,i.e.,notonlyatthecorrectpositionbutalsoatrˆ0+δrwhereδraccounts
forthecontinuityofrealspace,alimitednumberofmapneuronsaswellasaslightlocalization
uncertainty(Fig.4.7).Ourmodelcanreconstructqˆr0evenundernoisyconditionsandatslightly
wrongpositions(Fig.4.8);forexample,theﬁshbodiesofFig.4.1(seealsoFig.4.5)canbe
recognizedanddistinguished.Thusobjectlocalizationaswellasobjectrecognitionbasedonthe
estimationofmultipolesispossiblesimultaneously.
Inthesubsequentlineswediscussthelimitationsoftheproposedmethodandlikewisethe
theoreticallimitationofaquaticanimals’abilityofobjectrecognition.Tocomparethequalityof
diﬀerentmultipoleestimatesqˆatdiﬀerentdistancesd,wedeﬁneδq:=(qˆlm−qˆlm)2/qˆlm
asqualitymeasure.Hereδqlabelsthenormalizedstandarddeviationduetowronglyestimated
positionsrˆ+δrandnoise.Forsmallδqtheestimationerrorinqlmissmalland,inspiteofaslightly
wrongposition,theshapeparameterγcanberecognized.Forvaluesδq>1ashapereconstruction
basedontheestimatedqˆlmisimpossible.
Theresultsofournumericalexperiments,thatis,thelateral-lineperformanceourmodelpredicts,
supportthepreviousanalyticalﬁndingsasfollows.Moreprecisely,fortheoctupoleq30thecritical
distancewheretheerrorδqstartstogrowextremelyfastistheSMOsizeitself.Forlocalizationthe
criticaldistancecanbetakenasthelengthoftheDAlateral-linesystem,sayaﬁshlength,which
isincaseofapredatorhuntingforpreylargerthanthecriticallengthofshapereconstruction
4.9).(Fig.

4.Lateral-linePerception

qˆ30Figure(I−4.8:L)normalizedReconstructedbythedipoledipqˆole10(Astrength−D),q10forquadrupα=oleπ/qˆ220(andEv−Hariable),andshapoesctup1ole≤γ≤strength2at
withdiﬀerentnoiseatdistancesthed.mostThelikelygraypositioncolored(lighregiontgrey)depictsandtheatvslighaluetlyofincorrectlyestimatedmultipestimatedolepstrengthsositions
(darkgrey)asexplainedinthemaintextandcorrespondingtoFig.4.7.Sincewehaveusedan
approximationwithonlythreecomponents,thestrengthsofthemultipolesdiﬀerfromtherealqlm
(blacdistinguishkline).diﬀerenRemarktlyablyshap,qed30isobanjects.approWeseeximately(A,B,linearE,F,Ifunction,J)ofthatγforandthsmallusdialmoststancesbsuﬃcesetweento
ME.SMOIfdandDAapproacdiﬀerenhestlySMOshapsizeed(5bocm),diesthecanbereconstructiondistinguished,startsevengettinginspiteblurredofnoise(B,Cand,F,G,Jtruncated,K).
Atlargerdistances(D,H,L)nostrongcorrelationbetweenqˆ10,qˆ20orqˆ30andγcanbefound,and
thusnoshapecanberecovered.NeverthelessthetwoﬁshinFig.4.1with,say,γ=1andγ=2
canbedistinguishedatleastatd=3cm(A,E,andI)andd=4cm(B,F,andJ)bymeans
ofestimatingthewholesetofmultipolemomentsoronlythestrengthofqˆ30.Inbothcasesqˆ30
discriminatesγ=1andγ=2.

Insummary,ourmultipoleexpansion(ME)methodcanquantifystimuluscharacteristicsby
meansofreconstructingthemultipolemomentsqlm.Becausetheﬂowﬁeldneednotbegenerated
byasubmergedmovingobjectalone,themethodcanalsobeappliedtocharacterizecomposite
depictedsituationsinsucFig,has4.8schoandolingFig.orv4.9ortexwehavestructuresansweredfoundtheinthecompwakeellingofaﬁsh.question,Inpresenwhethertingatheresdetectingults
animal(DA)orartiﬁcialsystemcanlocalizeandrecognizeaSMOgivenonlythemeasuredwater
velocitiesattheDAlateral-lineorgansand,andifso,howwell.Thatis,theMEisusefulonlyif
theSMO,ordistanceshorbteetr.weentheOtherwiseobjectthe(SMEMO)reducesandtodetectoritsdip(DoleA)isapprosimpliﬁcation.ximatelyofthesamesizeasthe

erceptionPineLateral-l4.

Figure4.9:Thetwographsdepicttherelativereconstructionerrorδq30(dashedline)oftheoctupole
momentq30asdeﬁnedinthemaintextfordiﬀerentnumbersofneuromasts.Theyshowthat
fordistancesdlargerthanamovingobject’ssize(5cm)multipolereconstructionisnotpossible.
Thesolidlinedepictsthelocalizationerror||rˆ0−r0||,whichstartstogrowfastatadistance
comparabletothelengthofthedetectinganimal(10cm).Thisagreeswellwithexperimental
ﬁndingswherelocalizationperformancestartsdecreasingatdistancesbeyondtheﬁshbodysize
[54].Fordiscriminationtasks,ﬁshhavetobeveryclosetotheobjectunderinvestigation[39].The
leftgraphshowstheerrorsemploying250(left)and500(right)neuromasts.Asexpected,inboth
plotserrorsincreasewithdbutthediﬀerenceduetothenumberofneuromasts(withinbiological
marginal.isrange)

okoutloandDiscussion4.4Inthischapter,wehavefocusedontwomajoraspectsoflateral-lineperformance.First,nomatter
whethertheanimal’scapabilitytolocalizeprey,predators,conspeciﬁcs,orobstaclesisbasedonthe
processingofcharacteristicpointsofthevelocityproﬁleorthewholevelocityﬁeld,theSMOshape
playsanessentialrole[253].Assumingthatpotentialﬂowisanadequatetoolforapproximating
thecorrespondinghydrodynamics,wehavequantiﬁedanalyticallytheeﬀectofshapetogainfurther
insightsintomechanosensoryprocessing.Inadditionandfortheﬁrsttime,wehavepointedoutthat
diﬀerentlyshapedobjectsresultindiﬀerentboundaryconditionsforthehydrodynamicﬂowﬁeld
andthuscausedistinguishablemultipolemomentsqlm.Wehavedevelopedamultipoleexpansion
method(ME)thatpredictsthepossibleobjectrecognitionperformanceofthemechanosensory
sensorysystem–thelateralline.
Second,inthepresentliteraturethereisanaturalcharacteristiclength-scaleuptowhichthe
animalcanlocalizeobjects,viz.,itsbodylength,ormoreprecisely,thelengthofitstrunklateral
line[95,75].Here,wehavegivendiﬀerentlinesofevidencethatthereisasecondlengthscale
withinwhichshapeinformationistransmitted,namely,thebodylengthoftheSMOtobedetected.
IfthedistancebetweentheDAandtheSMOisbelowthatlengthscaleeventhesimplestobjects
transmitmeasurableshapeinformation.Thisinformationcanbeusednotonlytolocalizebuteven
torecognizehydrodynamicobjects[251].

4.Lateral-linePerception

Promisingstepsforfurtherenlighteningthisinterestingsensorysystemaremanifold.Inconcreto,
wesuggesttoexperimentallytesttheabilityofthelaterallinetorecognizeshapeindiﬀerent
settingstoevaluatethepredictedrecognitionperformance.Onecouldclarify,forinstance,the
numberofmultipolesthatisnecessarytorecognizeshapeinnaturalconditions.Anotherinteresting
aspectwouldbetoanalyzewhetherthesensorysystemis“blind”whenitcomestodistinguish
SMOswithspecialsymmetries.Moreover,thepossibleexperimentalsetupsmaynotbelimitedto
thelateralline,viz.,hydrodynamics.Air-dependentmechanosensorysystemssuchasthesensory
systemofcricketscanalsobeconsidered.
Onthetheoreticalside,itwouldbeveryinterestingtoextendourmodeltothetimedomain.
Thatis,ontheonehandmodelsthatenhancethepresentMEmodelshoulddealwithatemporal
varyingmultipolepatternq(t)tofaithfullyidentifyhydrodynamicobjects.Namely,themoving
bodyorthevorticesinthewakeofswimmingﬁshproducespeciﬁchydrodynamicstructuresthat
aretime-dependent,whichistrueforthecorrespondingmultipolesaswell.Presumably,the
time-dependentmultipolesareanimalspeciﬁc.Ontheotherhand,itispossibletoextendthe
presentMEmodeltonotonlyreconstructthemostprobableSMOpositionatonespeciﬁcpointin
timebuteventoestimatethemostprobableSMOtrajectory.Giventhetrajectoryandassumptions
onthemovingobjectsuchasitsinertmassitiseasytopredictitsfuturepositions.Thedipole
momentincludestheactualmovingdirectionandspeedoftheSMOandwouldthereforealready
with.starttosuﬃceInsummary,ourobject-recognitionmodelagreeswellwithbiologicalﬁndingsandprovidesa
theoreticalunderstandingofhydrodynamicobjectperceptionthroughthelateralline.Furthermore,
wenowunderstandthefundamentalrestrictionsforany(sensory)evaluationoflateral-linedata.
Finally,theseﬁndingscanalsobeappliedtobiomimetics[289];e.g.,toimprovepassivenaval
ms.systevigationna

Figure4.10:Sometimes,itmakesadiﬀerencetoknowwhatiswhere.Frommyposterpresentedat
the2009Sloan-SwartzSummerSchoolatHarvardUniversity.

ihrDieeGeWahrheitgnersterbtrienumphiertnuraus.nie,

5.InfraredSystemofSnakes

kPlancMax

Somesnakespossessaninfrareddetectionsystemthatisusedtocreateaheat-imageoftheir
chaenvironment.racteristicsInofthissensorychapter,en-andwedecoexploding,retheandinvolvedaddressthephysicsquesoftionsignalofhowmapping,theinfratheredneuronalsystem
cancreateafunctionalrepresentationofthethermalpropertiesoftheenvironment.

ductionIntro5.1Twogroupsofsnakes,pitvipersandboids,possessasensorysystemtodetectinfrared(IR)
radiation[198]whichallowsthemtoperceiveatwo-dimensionalimageoftheheatdistributionin
theirsurroundings.Thedetectionsystemconsistsofasetofcavitiescalledpitorgans.Inpitvipers,
apitorganislocatedoneachsideofthesnake’sheadneartheeyes;cf.Fig.5.1A.Suspendedin
eachcavityisaheatdetectingmembrane,whichissensitivetomKtemperaturediﬀerences[33,63].
TheopticalprincipleunderlyingthedetectionofIRradiationisthatofapinholecamerawithlarge
aperture;cf.Fig.5.1B.
Radiationenteringthroughthepitholehitsthemembraneatacertainspotdependingonthe
sourcedirection;seeFig.5.2A.Theresultingtemperaturechangeatthisspotisdetectedbyheat

Figurecourtesy5.1:of(A)GuidoHeadWofaesthoﬀ;pitviperadaptedwithfromnostril,[252].(large)(B)pitApihole,tvipander’seye,leftinfrared-sensitoright.tivepitPhotographorgan
wmemorksbrane,likeasusppinholeendedcamera.freelysoasRadiationtomiennimizeteringheatthroughlosstothetheopeningsurrounding(left)hitsatissue.Toheat-sensitivensuree
alargeenoughenergyinﬂux,theaperturehastobequitelarge(∼1mm),comparabletothe
distanceaperture-membrane.Theimageofapoint-likesourcethusformsadisc-shapedimageon
brane.memthe

InfraredSystemofSnakes

sensitivecells,freenerveendings(FNEs)thatareorganizedwithindensearchitecturescalledthe
terminalnervemasses(TNMs)[6,121].TheTNMsrepresenttheheatsensitivesensoryareasthat
aredistributedthroughoutthemembrane;cf.Fig.5.2B.Thereareabout40×40sensoryareason
themembrane[198],andtheﬁeldofviewisabout100◦wide,whichimplies◦thatinputtotheorgan
couldberepresentedinthebrainwitharesolutionofapproximately2.5.Sincetheradiationﬂux
enteringtheorganmustbelargeenoughtoquicklydetectmovingprey,theapertureoftheorgan
iswide,approximately1mm,andcomparabletotheorgandepth.Thusincomingradiationfroma
pointsourcedoesnotstrikeapoint-likeregiononthemembrane,asinanidealpinholecamera,
butrathermuchlargerdisc-shapedregion;cf.Fig.5.1.Itisnonethelesspossibletodetermine
thedirectionoftheincomingradiationiftheboundaryofthedisc-shapedregionremainsnarrow
enough.Thisapproachbreaksdownifmultipleornon-pointlikeheatsourcesarepresent.Thenthe
resultingheatdistributiononthemembranewillbeheavilyblurred(seeFig.5.10),precludinga
directevaluationoftheinput.
TheIR-detectionsystemofsnakespresentsaparadox.Theopticsofthepitorganaspinhole
cameraensuresthatimagesonthemembranewillbedistendedandblurry.Yettheinformationfrom
theIRsystem,combinedwithinputfromthe“normal”visualsystem,allowsformationofarather
preciseneuronalmap[116]inthebrain’soptictectum[109,210,211];cf.seeFig.5.2C.Thismap
issharpenoughtoserveasatopologicalrepresentationoftheoutsideworldinwhichneighboring
neuronsrepresentneighboringregionsoftheoutsideworld.Evenamultimodalinteractionbetween
theinfraredmapandthevisualmaphasbeenfound[109,210,211].Experimentalstudieshave
shownthatasnake’sorientationtoapointsourceofheatvariesbutthatanaccuracyof5◦is
neverthelessattainable[211],whichapproximatesourroughestimateabove.Wearethusfaced
withaparadox,inthattheopticalqualityofapitorganislowbuttheneuronalperformanceis
high.Howdoesthathappen?
TheparadoxinvolvestheopticalqualityofIR“vision”.Wethereforeexploretheoptical
qualityofthepitorganandaddressthethermodynamicphenomenainvolved.Thisenablesus
tocalculatetheheatdistributiononthemembraneforagivenheatdistributioninspace.Ina
secondstep,weaddressthequestionofhowthemappedheatdistributionisencodedneuronally.
Basedonthemembraneimageandtheknowledgeofsensoryencoding,wecontinuewiththe
thirdstepofsensoryprocessing,thatistoneuronallydecodetheoriginalspatialheatdistribution
fromthemeasurementsonthemembrane.Wedeveloptworeconstructionmodelswhichareable
toexplainhowtheneuronalsystemcancompensatetheoversizedpinhole.Sinceasnakehas
limitedcomputationalresources(all“calculations”mustberealizableinneuronal“hardware”)any
reconstructionmodelmustbesimple.Accordingly,ourmodelsuseonlyonecomputationalstep
(theyarenon-iterative)toestimatetheinputimageonlyfromthemeasuredresponseonthepit
membrane.BothresembleaWienerﬁlterandareakintobutdiﬀerentfromsomeofthealgorithms
usedinimagereconstruction[229].Ithastobeconstantlyborninmindthattheopticaldevice
discussedherehasverylimitedimagingcapabilitiesincomparisontohuman-deviseddetection
systems.Nonetheless,aswewillshow,itisstillpossibletoreconstructthespatialheatdistribution
inabiologicallyplausibleway.Inaddition,wequantifytheperformanceofourmodelsandgive
reasonablelimitationsforlocalizingwarmobjectsbymeansofthissensorysystem.Beyondvarious
speciﬁcissuesofthepitsthatareconsidered,weextendthecurrentunderstandingofthissenseby
meansofenlighteningthequestionofwhattheappropriatestimulustotheIR-detetctionsystemof
snakesis.Finally,wepresentanoutlookanddiscussstillopenquestionsthatshouldbeanswered.

InfraredSystemofSnakes

Figure5.2:Schematicdrawingoftheessentialpartsofthesnake’sIR-detectionsystem;adaptedfrom
NewmanandHartline[211].(A)depictsapitorganthatisilluminatedbytwoinfraredsources.The
illuminatedareas1and2onthepitmembraneheatup.(B)Sensoryresponsesaregeneratedwithin
thefreenerveendingsofthebranchingtrigeminalnerveaxonsthatformtheterminalnervemass
(TNM).Thespikesaretransmittedalongthetrigeminalnervetothenucleusofthelateraldescending
trigeminaltract(LTTD).(C)Withinthebrain,theinfraredinformationis,i.a.,representedinthe
optictectumbymeansofaneuronalmap.(D)Withintheoptictectumseveralcomputationalsteps
betweenthevisualmapandtheinfraredmapcanbefound.Theinformationofthesetwosensory
systemsfusesintoacomprehensiveperceptoftheenvironment.Foradetailedreviewofthesensory
pathwaywerefertoelsewhere[162,163,182,191,197,198,199,209,211,210,246,255,256,266].

5.2Warmingup:Signalmapping
Inthissection,wederiveequationsofhowtomapthestimulus,thespatialheatdistributionofthe
externalInitially,wweorldclarifyontowhicthehthesensorycellsappropriatewithinstimtheuluspitformemthebraneinfraredofthedetectionsnakeassystemdepictedisandinFig.whatits5.2.
characteristicsare.Insodoing,wederiveanappropriatemathematicaldescriptionofthesignal
cess.promapping

vistadynamicthermoA5.2.1Tostartwiththelatterissue,wefollowthewidelyacceptedopinionthatthepitmembranereceives
thermalradiationfromtheexternalworldandhenceheatsuporcoolsdown,viz.,itisawarm
receptororgan[48,49,168].Ontheotherhand,themembranedoesnotrespondtospeciﬁcwave
lengthsliketheretina[93,196].Wehencehavetoanalyzethemechanismsofhowthemembrane
heatsupandcoolsdown.Wederiveasimpliﬁedmodelofthecomplexheatingandwarmingprocess
toestimatetheimportanceofthediﬀerentmechanismsinvolved.
Attheoutset,weconceptuallydividetheheattransportphenomenaintotwocategories,the
heattransportwithinthepitmembraneandtheheattransferthroughthemembrane-airsurfaces.
Thatis,weconsiderheatconductionthroughtheair,energygainsandlossesthroughradiation,

InfraredSystemofSnakes

thebloodtempperfusion,eratureofheatasmallconductionpitmemwithinbranetheslicetissue,arescandhematicconvallyection.depictedAllintheseFig.pro5.3.cessesaﬀecting

Figure5.3:Sliceofthecenterofthepitmembrane.Wecanseethediﬀerentheatﬂowsthat
gomemthroughbrane-airthesurfacesdiﬀerenStma=surfaces,lp2.Thei.e,edgethemelengthmlpofbrane-memthebranequadraticsurfacessliceisS25mm×=10l−p6dmpandandthethe
thicknessofthemembranedpisassumedto15×10−6m.

Thestartingpointofourstudyisthebasicheatequationthatdescribestheheatdevelopment
withinthetissueofthepitmembrane.Itreads
ρpCp∂T(x,t)=kp2T(x,t)+Qi.(5.1)
t∂iInthissection,wedenotequantitiesassociatedwiththepitswithasubscriptp,forinstance,thepit
membranedensityρp(≈1080kgm−3).TheotherquantitiesareCp(≈3850Jkg−1K−1),theheat
capacitanceandkp(≈0.49Wm−1K−1),thethermalconductivity.Bothrefertothecharacteristics
ofthetissue.Asreasonablevaluesfortheparametersofthepitsarenotavailable,wehavetaken,for
thesakeofconsistency,valuesforneuronaltissue,i.e.,thebrain,fromFialaetal[72].Theﬁrstterm
intheright-handsideofequation(5.1),kp2Tp(x,t)describesheatconductionthroughthetissue.
Thefundamentalsolution,viz.,incaseQi=0isanexponentialdiﬀusionT(x,t)∼exp[−x2/ct]of
theheatproﬁlewithc=const..Theeﬀectofthistermisthatdiﬀerencesintheheatdistribution
getsmearedout[70].Wewillcomebacktothistermlateronand,insodoing,wespecifyand
discussthediﬀerentsourcesandsinksofenergyQi.
Thesecondtransportphenomenonwithinthetissueistheheattransportthroughblood.Many
experimentalﬁndings[6,92,121,205]havereportedadensecapillarysystemwithinthepit
membrane.Amemiyaetal.[5]evensuggestedthecapillariesfunctionasa“heatexchanger”;see
Fig.5.7.Pennes[222]ﬁrstformulatedandsubsequentinvestigatorsvalidated[50,72,207,286]the
so-calledbio-heatequationdescribingtheheattransportthroughthebloodQbforpartsofthe

InfraredSystemofSnakes

bodysuchasanarm,
Qb=ρbVbCb(1−κ)(Ta−T)=ωCb(1−κ)(Ta−T).(5.2)
HereQbistherateofheattransferperunitvolumeofthetissue.Theperfusionrateperunit
volumeofthetissueisdenotedasVb,ρbisthedensityofblood,Cb(≈4000Jkg−1K−1)isthe
speciﬁcheatoftheblood,κisafactorthataccountsforincompletethermalequilibriumbetween
bloodandtissue.Typically,itissetκ=0.ThetemperatureofthearterialbloodisdenotedasTa,
andTdescribesthelocaltissuetemperature.Despitethelargenumberofmitochondriaaround
theTNMs,wedonotassumeintheaboveequation(5.2)thatthepitmembraneitselfproduces
essentialthermalenergyaccordingtobiochemicalprocesses.
Next,weturntothephenomenathatappearatthemembrane–airinterfaces;cf.SmainFig.5.3.
Suspendedwithinthecavityofthepitorganthepitmembranedividesthepitorganintoaninner
andanouterchamber[9,198,211];seeFig.5.2.Inotherwords,themembraneissurrounded
byair.Sincewedealwithverysmalltemperaturesandtemperaturediﬀerences,aircannotbe
treatedasanidealinsulator.Theheattransportthroughtheaircanbedividedintoconvection
andconduction.Thepitmembraneisalmostperfectlyprotectedfromforcedconvectionbecauseof
itslocationwithinacavity.FreeconvectionwithinanenclosureonlyappearsiftheGrashofnumber
Gr:=gβL3(T−T∞)/ν21000[64].Hereg=9.81ms−2,βisthecoeﬃcientofthermalexpansion,
Lisacharacteristiclength,Tdescribesthetemperature,T∞thesteady-statetemperature,andν
denotesthekinematicviscosity.TheGrashofnumberdescribestherelationbetweentemperature
inducedliftingforceandviscosity.BecauseofthesmalldimensionsofthepitorganGr1000we
canneglectfreeconvectionasBakkenandKrochmal[9]havealreadypointedout.Withthehelp
ofFourier’slawweapproximatetheamountofheatenergylostbyconductionthroughairQaby
Qa=kaT−Tp+kaT−Tp=kadi+do(T−Tp).(5.3)
didodido
Heredianddodenotetheeﬀectivedistancesfromthepitmembranetothewalloftheinner(di)and
outercavity(do),respectively.Thethermalconductivityofairisgivenbyka=0.026Wm−1K−1,
andTpdenotesthetemperatureofthepitcavitywalls.
Theremainingtransportprocess,namelytheradiationprocess,isthemostimportantonesince
itupmapstheallmembrane,informationandofthistheistheexternalfocuswoforldthisontothesection.memThebrane.secondIntheproblemﬁrst–place,howradiationinformationheats
istransmittedandextractedbythepitorgans–willbediscussedlater.Wedescribetheenergy
transportduetoradiationprocessesQrby
Qr=−2σT4+µpσTe4+µoσTo4+σTp4(5.4)
bothsidesfrontsidebackside
energylossesexternalworldwarmobjectcavitywall
whereTdescribesthelocaltemperatureofthepitmembrane,asalwaysinthissection.The
meantemperatureofthesnake,viz.itscavitywalls,isdescribedbyTp,andTereferstothe
meantemperatureoftheexternalworldprojectedontothefrontsideofthepitmembrane.In
thefollowing,weassumeTe=Tpbecausethesnakeisacold-bloodedanimal.Thedimensionless
theparametersareaswithµpandmeanµotempdenoteeraturetheTpratioandtheunderwarmwhichatargetsmallobjectsurfacewithoftempthepiteraturememTo;braneobviously“sees”
µp+µo=1.Wecalculatedµo:=A/(2πd2)withA,thevisiblesurfaceofthewarmobject,and
thedistancedbetweenmembraneandobject.Wenotethattheinformationoftheexternalworld
iscontainedintheexactvaluesofµpandµoforeachsensoryareaonthemembranebutforthe

InfraredSystemofSnakes

heatingprocess;itsuﬃcestoassume,forinstance,µo≈0.025(d=0.5m)or0.1(d=0.25m).
Furthermore,weassumeallsurfacestobeblackbodyradiatorwiththeconstantσdenotingthe
Stefan-Boltzmannfactor(σ=5.67×10−8Wm−2K−4)emittingtheirenergyintoahalfspace.We
cansummarizethetermsandrewrite(5.4),
Qr=−2σ(T4−Tp4)+σµo(To4−Tp4)
Qr≈−8σTp3(T−Tp)+4σµoTp3(To−Tp).(5.5)
Itcanclearlybeseenin(5.5)thattheenergyinputandtheenergyoutputdependonheatcontrasts,
namely,theexternalandtheinternalheatcontrast.
Thepit-heatequationanditssimpliﬁcation.Wecannowcombinethediﬀerentsourcesand
sinksofheatenergywehavederivedaboveintoapitheatequationthatreads
T∂ρpCp∂t=kp2T+Qb+Qa+Qr.(5.6)
Givenallparameters,theaboveequationcanbesolvednumerically.Next,however,wefollowan
analyticapproachtoderivethecharacteristicsofanddependenciesupontheheatstimulus.
Aswehaveseenpreviously,alloftheQitermsdependontemperaturediﬀerencesinvolvingT.
Wecanreasonablysimplifythesituationbyassumingallpartsofthesnakeandthebackground
oftheexternalworld,exceptthepitmembraneandthetargetobject,tohavethesamemean
temperatureTp.Thisreduces(5.6)toanequationwithonlytwovariables,namely,theheat
contrastintheexternalworldξext:=To−Tpandtheinternalheatcontrastξint:=T−Tp
onthepitmembrane.Furthermore,wecanidentify∂ξint/∂t=∂T/∂t.Inaddition,weleave
thediﬀerentialdescription(5.6)andconsiderasmallpieceofthepitmembranelocatedinthe
centerofthemembranewithasquaredbaseoflp×lpandathicknessdp;cf.Fig.5.3.Wetake
lp=25µmanddp=15µm.Thelengthlpcorrespondsapproximatelytothesizeofasingle
independentreceptorarea(terminalnervemass).Figure5.3depictstheheatﬂowsthroughdiﬀerent
surfacesofthemembranecuboid.Themembrane–airsurfacesareSma=lp2=625×10−12m2,the
membrane–membranesurfacesareSmm=lpdp=375×10−12m2,andthevolumeofthecuboidis
Vp=lp2dp=0.01×10−12m3.Thesimpliﬁedpit-heatequation(5.6)foroursmallcuboidnowreads
τp∂∂ξtint=−Qoutξint+Qinξext,(5.7)
whereτp=ρpCpVp(≈40×10−9JK−1).WecanestimatethevaluesofthetermsQoutandQinby
consideringthecorrespondingmembranecuboid;cf.Fig.5.3.
3di+do4k
Qout=Smaσ8Tp+Smakadido+Smmrˆp
≈8×10−9WK−1≈15×10−9WK−10−10×10−6WK−1
+ωCbVp25×10−9WK−1
≈0.4×10−9WK−1
1Qin=Smaroσ4Tp3100Qout(5.8)
≈0.1−0.4×10−9WK−1

(5.8)

InfraredSystemofSnakes

Thestaticpit-heatequationand“optical”contrast.Wenowset∂ξint/∂t=0in(5.7)and
computesimilarlyto[9]themaximalvalueoftheinternalheatcontrastξmaxandthesteady-state
relationbetweeninternalandexternalheatcontrast
ξmax=Qinξout1ξextandξmax1.(5.9)
Qout100ξext100
Wecanseeclearlyfrom(5.9)thatthequalityofthe“opticalsystem”pitorganispoor,i.e.,it
representsexternalcontrastsroughly100timeslessstronglythantheyareintheexternalworld.
Inotherwords,todetectobjectswithabout1Ktemperaturecontrast,snakeshavetomeasure
temperaturecontrastonthepitmembraneintheorderof10mK.Thispurelytheoreticallyderived
resultcorrespondsto,andexplains,manyexperimentalandbehavioralﬁndings[9,33,63,211].

Figure5.4:Thesteady-stateinternalheatcontrastξmax.Abovewehavestudiedthesteady-stateof
theinternalheatcontrastξmaxofasmallsliceinthecenterofthepitmembrane(seeFig.5.3)by
meansofvarying(A)theexternalheatcontrastξext,(B)thedistancebetweenmembraneanda
warmobjectdMO,(C)thebloodperfusionrateofthemembraneωb,and(D)thedistancebetween
theinnercavitywallandthebacksideofthepitmembranedICM.Exceptifstatedotherwise,
parametersareQin=0.4×10−9WK−1,Qout=50×10−9WK−1,andξext=5K.

discussTogettheatermsdeeperin(5.8).understandingInitiallyof,thetheimpheatortanceconductionofthetermdiﬀeren∼tgradeﬀects,Tiswsetrnextonglyanalyzesimpliﬁedand
to∼conduction1/rˆp.Ifwithinweusethethistissuewouldsimpliﬁcationthenandclearlytakerˆdominateptoballetheotherradiusterms.oftheThispitapproacmemh,branehowevheater,
istoosimple.Foranumericalcalculationrˆpreferstothedistancelpbetweenthecenterofthe
cuboidandtheneighboringmembranecuboids.Inrealsituations,theneighboringcuboidswill
butshowalmaostsimilarzero.tempNeveratuertheless,reasinthetheobservcaseedthereone.isanHenceexternalthetempheateraturesource,thediﬀerencesradiatedwillheatnotbeenergyξint
willgradienheattsupatthethememedgesofbranethenotilluminatedhomogeneously;memcf.braneFig.area5.10.thatThewillwbearmingerasedproquiccessklygbyeneratestheheatheat
conductionterm.Inabsenceofsteeptemperaturegradientsonthepitmembrane,i.e.,between
tworeceptorareasinthecenteroftheilluminatedmembranearea,wecanneglectthemembrane
term.conductionheatdepTheendsseoncondthemaanatomjorytermofoftheQpitoutinorgan.(5.8)Thedescribgreateresheatthedistanconductioncebetweenthroughthetheinnerair.caThivitsywtermall

InfraredSystemofSnakes

andthebacksideofthepitmembranethesmallerthistermandthebettertheheatcontraston
themembrane[9];seeFig.5.4D.
ThenextimportanttermofQoutin(5.8)isduetoradiationlosses.Amemiyaandothers
[4,38],forinstance,havereportedaspecialsurfacestructureofthepits.Itseemstobeplausible
thattransmittanceandreﬂectioncouldbeminimizedthroughsuchasurfacestructure.However,
conservabsorbance.ationSiofnceenergythey,aredescribassumededbtoybKirecnearlyhhoﬀ1’s[9la]w,wesetequalsboththethermalthermalradiationradiationemittanceemittanceand
andtheabsorbanceto1.Accordingly,thepossiblevariationofthesetwoparametersisverysmall.
Moreover,experimentalquantiﬁcationismissing.
brainThetissuelast[72],termthisofQtermoutisdescribquiteessmalltheheatandlossesunimportanthroughtasbloweodhavﬂoew.IfdepictedwetakineFig.the5.4valuesC.Bufort
asreportedinmanyexperiments,themembraneshowsagreatamountofcapillariesanditisof
verysmallscalecomparedtothetissueusedforderivingthebio-heatequation[222].Moreover
andunfortunatelywithoutgivingnumbersontheinterestingtimescale,Gorisandcolleagues
[92,121,205]havereportedthatthebloodﬂowwithinthecapillariesfollowsthetemperature
development.Theyalsospeculatedaboutare-coolingmechanismoftheTNMbymeansofthe
bloodﬂow,whichevenseemstobecontrolledlocallybytheTNM.Accordingly,weeitherhaveto
modifytheequationbymeansofconsideringω=ω(T)(seeFig.5.4C)orwehavetomodelthe
eﬀectofbloodﬂowotherthanPenne’sbio-heatequationbecausethedimensionsofourproblem
aremoretwo-dimensionalinsteadofthree-dimensional.
Ontheotherside,Qinandthusξmaxisinﬂuencedbytheexternaltemperaturecontrastξextas
wehavedepictedinFig.5.4A.Thegeometryofthesetting,inmathematicaltermsµo,isinﬂuenced
bythesizeofthevisiblesurfaceAandthedistancedMObetweenthemembraneandthewarm
objectofinterest.ThedistancedMOlimitsthemaximalheatcontrastξmaxandthusthedetection
rangeaswehaveillustratedinFig.5.4B.
Thetime-developmentofthepit-heatequation.Anotherimportantaspectof(5.7)isthe
timedevelopmentoftheheatcontrastandthusatheoreticalapproachtotheresponsetimeofthe
infraredsystem.Wecanrewrite(5.7)tothediﬀerentialequation
∂∂ξtint=−τ1ξint+Qˆin,(5.10)
wherewehavesubstitutedτ:=τp/QoutandQˆin:=Qinξext/τp.Thegeneralsolutionoftheabove
equationisgivenbyasuperpositionofthehomogeneoussolution(Qˆin=0)andaparticular
solution.Combined,thesolutionreads
ξint(t)=ξmax1−exp−t,(5.11)
τwithξmax:=τQˆin=(Qin/Qout)ξext–theresultforthesteady-statecondition(5.9).Underthe
assumptionsandsimpliﬁcationswehavemadeabove,wereceiveatime-dependendequationthat
allowsustostudythequestionofwhatinﬂuencesthewarmingprocessofthepitmembrane(5.11).
InFig.5.5wehaveillustratedthedependenciesofthewarmingprocessupontheinﬂuencing
variables.Asaspin-oﬀweﬁndphysicalevidenceforexperimentalresults[33]thatsupposea
tempneuronaleraturerespcononsetrastbeloonwthe100msmemusingbrane,mKisnottempconstaneraturetduringresolution.thatpTheeriodsignal,oftimehowbutever,i.e.,increasing.the
Inthissection,wehavefortheﬁrsttimederivedanequationincludingallaspectsofenergyin-
andoutputofthepitmembranewarmingprocess(5.6).Bysimplifyingthisequationto(5.7)we
havequantiﬁedboththedependenciesoftheheatcontrast(5.9)andthetime-dependentwarming
process(5.11).Thusweareabletoidentifytheimportanteﬀectsandshowwhichdataarecurrently
missing.Fromtheexperimentalpointofview,wehavearrivednotonlyatatheoreticalvalidation

InfraredSystemofSnakes

Figure5.5:Transientwarmingprocessofthepitmembrane.Abovewehavestudiedthewarming
ofasmallsliceinthecenterofthepitmembrane(seeFig.5.3)bymeansofvarying(A)the
externalheatcontrastξext,(B)theamountofenergylostbythemembraneQout,(C)theamount
ofenergyreceivedbythemembraneQin,and(D)parametersofthepitmembraneτpincludingfor
instancethemembranethickness.Thebasicgraph(red)correspondsinallpanelstotheparameters
Qin=0.4×10−9WK−1,Qout=50×10−9WK−1,τp=40×10−9JK−1,andξext=5K.

and,moreimportantly,ataclariﬁcationofthepossiblesensoryperformancesbuthavealsoprovided
mathematicaltoolsforfurtherinvestigations.
Inthenextsection,weturntotheinformationcontentthatistransmittedtothepitorgan.
Thatis,wewillclarifytheradiationprocessinmoredetailbymeansofdisentanglingµoforevery
sensoryareaonthewholemembrane.

heattheMapping5.2.2Haandvingthattheclariﬁedaccuracythatonlyofthetheheatradiationconoftrasttheplaysexternalacritiwcalarmrole,objewctsehacanvetocauseaanalyzereasonablecarefullystimwhiculush
inphysicaltegratedspdescriptionectralﬁtsradiancebestu(toλ,Tm)ap=πtheI(λ,T)externalderivheatedbyconMaxtrastPlantocktheinthe1901pit[226]membrane.The
u(λ,T)=2hc52πhc1Js−1m−3,(5.12)
λexpλkT−1
describestheamountofenergyablackbodywithtemperatureTemitsintothehalfspaceper
time,perwavelengthλ,andperunitsurface.AswecanseeinFig.5.6,almostall(99,5%)ofthe
tenthradiatedoftheenergytypicalfrompitablacorganskbapodyerturewith[47T,=137303].KWeisecanmittedthereforebelowignore100µmeﬀectswhichthatisareaboutdueoneto
thethatwais,vethecharactermappingofoflighthetandspatialinsteadheatusedistribgeometricution.Asopticsatoside-remark,describethephatmosphericysicsoftranoursmittanceproblem,
canbeignoredoverdistancesofmeters[8];sowecanneglectthiseﬀect,too.

Next,wecondensethephysicsofgeometricalopticsintoaformalismthatmapstheemitted
signalsfromtheexternalenvironment,forinstance,apreyontothepitmembraneofasnake.In
theprevioussubsection,wesawin(5.7)thattheexternalheatcontrastξextisanappropriate

InfraredSystemofSnakes

Figure5.6:Thespectralradianceu(λ,T)(left)andthenormalizedintegralofu(λ,T)(right)asa
functionofthewavelengthλattemperaturesofT=303K.Weseethatthewavecharacteroflight
canbeignoredandwecanusegeometricoptics.

descriptionforthestimulusbecauseQin–aconstant,mainlygeometricfactor–timesξextisthe
receivedenergyonthemembrane.WenowleavethesimpliﬁedQin,whichisadescriptionforone
areaofthepitmembranebymeansofµoandµpin(5.5),andderiveadescriptionforallsensory
areasionthepitmembraneaccountingforanarbitraryspatiallydistributedexternalheatcontrast.
WethereforediscretizetheexternalspaceinfrontofthepitintoplainsurfacesjofequalspaceSext
withtheheatcontrastξext;jfromnowondenotedbyξj.Usingthegeneralformalismderivedin
Chap.2,thereceivedenergyriduetoradiationofasensoryareaionthepitmembranenowreads
ri=hijξj.(5.13)
Themappingismathematicallymodeledbymeansofthetransferfunctionhijcomponent-by-
componentwherethecomponentsofthetransferfunctionare
jSmaSextσTp31/d2ijcos(ϕ),ifiisvisiblefromj
hi=0,otherwise.(5.14)
Toaccountfortheconservationofenergy,theemittedenergyscalesas1/d2ijcos(ϕ)wheredij
denotesthedistancebetweenthesurfacesjintheexternalworldandionthemembrane.Werefer
toϕastheanglebetweenthenormalvectorsofthesurfacesiandj,andthereforejthecosineterm
compensatesfordiﬀerentorientationsofthetwosurfacesiandj.Thetensorhiisonlydetermined
bythegeometryofthepitorgansandthemeantemperature.Wecanassumebothtobeconstant.
So,thetensorprojectseveryheatdistributionoftheexternalworldinthesamewayontothepit
brane.memToexplicitlycalculatetheresultinginternalheatcontrastonthemembrane,onehastosolve
(5.7)foreachsensoryareaigivenri,oronecouldlinearize(5.7)andjustmultiplyribythe
appropriatefactorderivedfrom(5.7).Aswewanttostudyinformationcontents,wewillusethe
pureheatintensitydistributionriassensoryinputinthefollowing;cf.Chap.1.
Forrealisticallymodelingthemappingprocessoftheheatintensitydistributionontothesurface
ofthesensorycellsiofthemembrane,wealsohavetoconsideruncertaintiesandnoise.Two
sourcesofnoisecanbeidentiﬁed.Noiseofthesignalandmeasurementerrorsofthedetector,
respectively.Wemodelbothbymeansofaddingstochasticvariablesαjandχitotheinputand
thedetectorresponse
ri=hij(ξj+αj)+χi.(5.15)
Thenoiseofthesignalcanbecausedbymeansofdisturbancesofotherobjectssuchasgrassand
fastmovements.Sinceeverydetectionprocessisblurredbynoise,itisreasonabletoassumethe
samehere,too;cf.Chap.2andChap.3.

InfraredSystemofSnakes

Insummary,wehaveidentiﬁedtherelevantphysics,i.e.,geometricaloptics,andderiveda
modelthatdescribestheheatintensitydistributionacrossthepitmembrane.Inthenextsection
weturntothedetectionprocessitself.

Figure5.7:Capillarynetworkandterminalnervemass(TNM)ofthepitmembrane;adapted
from[5,6].Theupperpanels(A−B)depictscanningelectronmicrographsshowingthecapillary
vlosingasculaturetheirofsheaththepit(smallmemarrobrane.w)andThe(Dlo)wtheerfreepanelsnervilleustrateendings(C)(nf)theofmayeterminallinatednervnerveeﬁbmassersarra(mfy).
Acollectionoffreenerveendingsformsareceptorunitandrespondstotemperaturestimulias
modeledinSec.5.3.Experimentaldataonthenervous-controlledcapillarynetworkanditseﬀect
onthewarmingprocessaremissing.

dingencorySenso5.3Inthissectionwewanttoslightlytouchuponthequestionastohowthestimulusisencoded
neuronallytoextractimportantideasfortheoverallfunctioningandperformanceoftheinfrared
es.snakofsystemdetectingFreenerveendings–amulti-purpusedetector.Weknowthatthelocationwherethe
stimwithinulustheisfreedetectednerveistheendingspit(FNE)memofbrane.theMoreterminalinnervparticularemass,the(TNM)sensoryofpitrespmemonsesbarerane.Ingeneratedother
words,thefreenerveendingsencodethetemperatureoftheirsurrounding(staticresponse)and
theyfoundinrespondothertoanimalstempersucaturehascmice,hangesvampire(dynamicbats,resporonscatse)[[3324,,117118].,F243ree,244nerv].eInendingsthesearesettings,also
vFNEsalidit[y.41]First,areknothewnstaticasresp(mammalian)onsecurvcoldesorw(neuronalarmreceptors.activityvs.Twotempﬁndingserature;seemcf.tobeinlaofyunivFig.ersal5.8)
ofandthe[24,235thermo].Second,receptorsinttohediﬀerenanimalstlevaboelsveof–constanalthoughtnottempaseraturepronouncedareassimilar;inthecomparesnake’s[117pit]

InfraredSystemofSnakes

membrane–thereisalsoadynamicresponsetotime-dependentthermalstimuli;cf.Fig.5.8and
e.g.[243,244].InpassingwenotethatFNEsseemtobeusedalsoinspeciﬁcwatersnakesfor
detectingwatermovements[43].

Figure5.8:Averageimpulsefrequencyofapopulation(N=9)ofwarmﬁbersfromthetrigeminal
branchwhenapplyingthermalstimulitotheskinofaBoaConstrictor;adaptedfrom[117]and
[25].Duringthisexperiment,Henseletal.haveincreasedthetemperature(redcoloredareaand
righty-axis)step-wise(ΔT=5◦C),startingatT=20◦CandendingatT=40◦C.Aswecan
see,themaximaofthedynamicresponseaswellasthesteady-statevaluesofthestaticresponse
(seeinlaygraphC)followabell-shapedcurve.Theinlaygraphshowsthesteady-stateresponse
curvestostatictemperaturestimulusof(A)acat(N.lingualis,bursting),(B)adogﬁsh,(C)aBoa
Constrictor,and(D)acat(N.infraorbitalis,non-bursting).Notonlytheresultingdetectoractivity
isqualitativelycomparablealsotheunderlyingdetectionprincipleseemstobethesame:freenerve
endings.

Amodelforthermoreception.Thestaticresponsecurvesofmammalianthermoreceptors,the
so-called“coldﬁbers”,andtheunderlyingbiologicaldetectionmechanism–freenerveendings–seem
tobethesameoratleastcomparabletowhatweﬁndinthepitmembrane;cf.Fig.5.8.Wehave
thereforeadaptedacommonionicmodelformammaliancoldﬁbers,theHuber-Braunmodel[26,27,
28,124],tostudyFNEresponses.Notsurprisingly,wecanreproducetheexperimentallyobserved
staticresponses,abell-shapedfrequencyvs.temperaturecurvature.Evenmoreinterestinglyand
fortheﬁrsttimewehavebeenabletoreproducethedynamicresponses;seeFig.5.9.Toour
knowledge,nootherthermo-receptormodelhasbeenabletodoso.
TheHuber-BraunmodelforthermoreceptorsisbasedonasimpliﬁedversionoftheHodgkin-
Huxleymodel.Themodelconsistsoftwosetsofionicconductancesthatoperateattwodiﬀerent
voltagelevelsandtimescales.Thetemperaturedependenceismodeledbytwoscalingfactorsfor
therateconstantsρ(T)andthemaximumoftheconductancesΦ(T).Themainobservableisthe
membranepotentialVthatiscontrolledbyﬁvecurrentsaccordingtothemembraneequation
VdCMdt=−INa−IK−Isd−Isr−Il,(5.16)
whereCMisthemembranecapacitance.Theioniccurrentsoftherighthandsidecanbegrouped
intothreecategories.TheﬁrsttwofastcurrentsINa/Kgeneratetheactionpotentials.Thenext

InfraredSystemofSnakes

twocurrentsIsd/srareslow,subthresholdde-orrepolarizing,andtheirinterplaycausesmembrane
potentialoscillations.Finally,thelastcurrentIlisthepassiveleakcurrent.Allcurrentsare
modeledinthefollowingway
Ii=ρgiai(V−Vi),(5.17)
withρatemperaturescalingfactor,githemaximumconductance,aitheactivationvariable,and
Vithereversalpotential.Theactivationvariableaiitselfcanbedescribedbyadiﬀerentialequation.
ForadetaileddescriptionoftheHuber-Braunmodelanditsnumericalparameterswereferto
elsewhere[26,27,28,124].Inthelightofourfocus,theHuber-Braunmodelcanreproducethe
neuronalresponsestostatictemperaturelevels,e.g.,thebell-shapedactivityvs.temperature
characteristic;seeFig.5.8.Yetthemodelcannotreproducetheobservedneuronalresponsesto
eratures.ptemhangingcTheHuber-Braunmodelanditsextension.Wemodifytherepolarizingsubthresholdcurrent
Isr=ρgsrasrf(V,T)in(5.17)(correspondstoequation(5)in[27])byintroducingatemperature-
gradient-dependentactivationvariableasrthrough
asr(t>t0)=asr(t)−ΔasrifgradT|t=t0=0.(5.18)
Thismeansthatifatemperaturestepoccurs,theactivationvariableasrisreducedbyΔasr=0.5.
Itismathematicallyunimportantwhetherwemodifyasrortheconductancevariablegsr.The
resultisthatIsrisreducedshortlyafterachangeintemperature,andtheneuronﬁresaburst
ofspikes.Thismodiﬁcationcorrespondsinprincipletoatransientionicgatingmechanismdue
to,e.g.,thermo-sensitiveTRP(transientresponseprotein)channels[23,176,189,212,275]which
resultsinanadditionalcurrentoccurringattemperaturechanges.Bothatemperature-depending
ionicgatingmechanismbymeansofTRPchannels[97]andanadditionaltransientcurrentsensible
fortemperaturechanges[219]havebeenfoundinthesnakerecently.
TheresultofournumericalexperimentisdepictedinFig.5.9andagreeswellwithbiological
ﬁndingsofthesnake’sneuronalresponses;cf.Fig.5.8.Thatis,theextendedHuber-Braunmodel
cannowreproducetheneuronalresponsestotemperaturechanges.Moreprecisely,ourﬁndings
hinttowardsthreeimportantissues.
•First,evenifthemechanismofthedynamicanswerismuchmorecomplex–aswebelieve–
itcanbeunderstoodatﬁrstsightbymeansofionicmodelsincludinganadditionalthermo-
gradient-sensitivecurrentorchannel.
•Second,wewanttohighlightthatatemperaturechangemodeledinthewaydescribedabove
synchronizesthediﬀerentFNEs.WepredictthatsynchronizingeﬀectsofFNEresponses
willenhancereliability,justasinmanyotherneuronalsensorysystems[80,81].Aprecise
quantiﬁcation,however,willneedamoredetailedanalysis.
•Third,wecanseeaprominentdiﬀerenceinthequalityofthestaticandthedynamicanswer.
Thetemporalprecisionandthesuddenchangeinfrequency(≈80Hz)atatemperaturestep,i.e.,
thedynamicanswer,ascomparedtotheincrease(≈5HzforΔT=5◦C)inthecorresponding
staticresponseisveryhigh.Inotherwords,wecandeﬁnetherelativequalityΩofhowprecisely
theFNEssystemcanencodeatemperaturechangebymeansofachangeofthecorresponding
neuronalactivityfromATatTtoAT+ΔTatT+ΔT,

Ω=AT+ΔT−AT.
AT

(5.19)

InfraredSystemofSnakes

Figure5.9:SimulationofFNEresponses,thesensoryencodingstep.Theneuronalactivityof
uncoupledneurons(N=26),theaveraged(timeintervalsare200ms)ﬁringrate,isplottedon
the(left)y-axis.TheneuronsaremodeledaccordingtothemodiﬁedHuber–Braunmodelof
thermo-sensitivecoldreceptors[27,28,124]asdescribedinthemaintext.Theneuronssequentially
start50msaftereachother.Duringournumericalexperiment,wehaveincreasedthetemperature
(redcoloredareaandrighty-axis)step-wise(ΔT=5◦C),startingatT=5◦Candendingat
T=30◦C.Aswecansee,themaximaofthedynamicresponseaswellasthesteady-statevaluesof
thestaticresponsefollowabell-shapedcurvecomparabletothebiologicaldata;cf.Fig.5.8.The
x-axisispresentedpiece-wise(e.g.fromt=t1tillt=t1+5s)aroundthetemperaturestepsbecause
thetemperaturestepssynchronizetheoutputoftheFNEsandpreventareasonableaveraging.We
concludethatthemodiﬁed(seemaintext)Huber–Braunmodelcanreproducethesensoryencoding
(FNEsresponses)oftemperaturedynamics.ThatisTRPchannelsresultinginasub-threshold
currentsthataresensitivetothermalgradientscancausethedynamicresponse.

Wecannowseethatthedynamicanswerencodestemperaturechanges:temporallypreciselyandvery
reliably;Ωdyn≈80Hz/20Hz=4.Incontrast,thestaticresponsetoabsolutetemperatureislesspre-
cise,relativelyweak,andevenambiguousontheleveloffrequency;Ωsta≈5Hz/20Hz=1/4=Ωdyn/16.
Thisbecomesevenmoreevidentifweadditionallyconsidertheamountoftimeinwhichthechange
ofactivityΔAoccurs.

encodedbAccordinglyythe,wstaticecanrespsponseeculatebutthatinsteadtheusessnakedochangesesnotofusethethetempabsoluteeratureteconmptrasteratureencocondedtrastby
thedynamicneuronalresponse.Thepresentparagraphshaveshownthatthedynamicanswerand
yettempboralelievced.hangesNext,ofwethewillheatexploitcontrasttheonresultstheofmemthebranepresenpttossiblywoplasections.yamoreThatimpis,weortancomtrolbineethanour
knowledgeonthephysicaltransmissionprocessandtheﬁndingsofthesensorydetectionprocessto
deriveaplausiblemodelfortheneuronalreconstructionmechanism.

InfraredSystemofSnakes

dingdecoNeuronal5.4Inthissection,westartwiththesimplestmodelwecanimaginethatisabletoreconstructthe
heatcontrastdistributionξintheexternalworld.Thisminimalmodelaccountsforthelimitation
ofaneuronalsystembymeansofrestrictedcomputationalpower.Weextendtheminimalmodel
bymeansofincorporatingpropertiesofthesensoryencodingprocesssoastoarriveataplausible
andmeaningfuldescriptionoftheinfrared-visionsystemofsnakes.Inaddition,wetestourmodels
invariousconditionswithregardtoexternalheatcontrastandomnipresentnoise.Finally,we
showthatandpredicthowfeatureextractionispossibleusingboththestatic(minimalmodel)and
dynamic(extendedmodel)responsesoftheinfrareddetectorsofthepitmembrane.
delmoMinimal5.4.1Wehavetothinkofsimplemodelsbecausethesnakehaslimitedcomputationalresourcesinview
ofproptheosefactamothatdelallthatiscalculationsnon-iterativmustebandeusesrealizableonlyinoneneuronalcomputationalhardwarestepandthatfast.canbWeethdoneerebfyorea
feedforwardnetwork.Thereconstructedexternalheatcontrastdistributionξˆjatlocationjcan
thereforebewrittenasalinearcombinationofthemeasuredintensitiesri,
ξˆj=ljiri.(5.20)
Theoptimalreconstruction.Toﬁndtheoptimalcoeﬃcientsofthereconstructiontensorlji
wefollowtheframeworkofoptimalmapformationpresentedinChap.2anddeﬁnetheexpectation
valueEofthequadraticerrorofourestimateξˆias
E=(ξˆi−ξi)(ξˆi−ξi).(5.21)
Intheaboveequationweinserttheexpressionξˆi=lijrj=lijhjk(ξk+αk)+χjfrom(5.15)and
(5.20).Furthermore,wecomputetheexpectationvalueoverallpossibleinputαianddetector
χinoises.Thenoise-freepartoftheinputξiisdeterministic.However,wewanttooptimally
reconstructnotonlyonespeciﬁcsituationbutthetypicalIRenvironment.Inotherwords,
biologicallyrelevantsignalsbelongtoaclassofsignalsthatwedenoteby“typical”.Consequently
aspeciﬁcsensorysignalisaconcreterealizationofatypicalbiologicallyrelevantsignal;henceit
isastochasticquantity.Wethereforehavetodealwiththeexpectationvalueofthesignal;fora
detaileddiscussionofwhywehavetotaketheexpectationvaluewerefertoSec.2.2.2.
ibeInzerothbeecausefollowingthenoisecalculationstermsofareδEindep/δhjenden=0,tofweeachassumeotherallandappofearingtheinputcrossstrengths.correlationsTheto
correlationtermscanbeapproximatedtoﬁrstorder
χiχj=σχ2δij,αiαj=σα2δij,andξiξj=σξ2δij(5.22)
whereδijdenotestheKroneckerdelta.Allthreeequalitiesdeﬁneautocorrelations.
Theﬁrsttwotermsdenotethenoisetermsontheinputandthereceptormanifold.Herewerefer
tonoiseasaspatiallyindependentadditiverandomvariablewithzeromeanandstandarddeviation
σχ/αthatmodelsthefollowingpoints.Evenifthedetectionmechanismonthepitmembraneis
thefast,asignalsignalwillinbetheavexterneragedaltowtheorldrelevcanbanetfaster.signalSothatwecancanbedeﬁneanreconstructed.integrationThislowindow-passwwithinﬁlter
eﬀectisinherenttoanysensorysystem,nomatteriftechnicalorbiological.Theinputnoiseαi
accountsforthiseﬀectandthereforeblursthetruesignalξi.Strictlyspeaking,thenoisewould
thenbespatiallycorrelatedduetoacommonmovementoftherealIRobject.However,wewill

InfraredSystemofSnakes

Theignoresecondthisknonoisewledgetermanddescribthusesdealtheinthedetectorfollonoisewingonwiththeawpitmemorst-casebranescenariothatofmayknowingoriginatenothing.from
physiologicalprocessesintheheat-sensitivecells.
Thethirdtermin(5.22)describestheautocorrelationoftheinputthatwehavemotivated
abovdeviatione.Inσofgeneral,theitinputcanξb=e(.w.r.,ξitten,..as.);ξiseeξj=Chap.(µξ22.+Wσeξ2)δwillijonwithlymuseeanoneµξtermandconthetainingstandardthe
iξtnowypicaldependsinformationontheabpoutersptheectivIReofsurrounding,whethertheviz.,relevanteitherthestimulusmeanortheinformationstandardisdeﬁneddeviationas.theIt
te.g.,ypicalpreyh.eatThecontrastformerorcaseastherefersvtoariationtheofmeanatµξypicalandheattheconlattertrastcasetorelatingthetostandanobardjectofdeviationinterest,σξ.
Forwithoutthesaklosseofofuniformitgeneralityybinyσourξandskipmathematicalforthetimedescription,beingwecathenabodenotevethequestionimpofortantwhatquanistitthey
relevantstimulustothepitorgan.Indoingso,weassumethatthecorrelationbetweeninputsat
diﬀerentlocationinspacei=jvanishes.Thatis,aworst-caseansatzagainbecauseknowingthe
valueoftheinputatoneparticularpointnowdoesnottellussomethingaboutitsvalueatother
pfromositions.thesameThisobjectassumptionortheofsamecoursewillsurrounding.notholdHeatinrealittranspybortecausephenomenaneighboringpeliminateointstesteepndthoebate
howgradienever,ts,lessandthantheptempossibleeraturemakleveselofneighreconstructingboringthepoininputtswillonlymorethereforebediﬃcult“similar”.andsuﬃcesAssuming,fora
h.approacminimalHavingclariﬁedthemappingoftheinputontothepitmembrane(5.15)andhowwecandescribe
ljtheiasnoisedeﬁnedandininput(5.20).cThatharacteristicsis,weha(5.22),vetowenextminimizeprotheceederrorﬁnding(5.21)thewithoptimrespalecttoreconstructionthecoeﬃcientensorts
ofthereconstructiontensor,i.e.,δE/δhji=0.Deﬁningthedimensionlessparameterσ:=σχ/σξ
andτ:=σα/σξwecancomputethecomponentsofthereconstructiontensorljisoastoﬁnd
likhjvhkv(1+τ2)+σ2δjk=hji.(5.23)
toNextrealwequantakeatities.briefThelookatparameterthetwτomodenotesdeltheparametersinverseτandσ,signal-to-noisetheirmeaning,ratioofandthetheirinputwhicrelationh
ingeneralwillbesmall.Thesecondparameterσrelatesnoiseonthepitmembranetothemean
vki.e.,inputitintenincreasessityinitstheeigenvexternalalues,wtheorld.valueSinceofσσdirectlyinﬂuencesmothediﬁesthereconstructiondiagonaloftensorthematrixsigniﬁcan(hjtlyh.v),

InfraredSystemofSnakes

Figure5.10:Minimal-modelreconstructionperformance;ﬁguresareadaptedfrom[252].We
discretizeAlbrechtD¨urer’s1502paintingofahare(left)intoapixelizedversionwithan8-bitgray
scaleintendedtorepresentthehare’ssurfacetemperature.Inacomputermodel,thisIR-source
imagewaspresentedtoanidealizedsnakepitorgan.Thedashedcircleindicatestheorgansﬁeldof
view(100◦).MappingtheIRsourcebymeansofthetransferfunctionhgivenby(5.14)ontothe
pitmembraneresultsinaheavilyblurredmembraneheatimage(middle).Inthenextstep,the
modelreconstructstheIRinput,i.e.,theharefromtheIRimageonthemembraneofthemodelpit
organ.Therightpanelsshowthereconstructedimagebymeansofthereconstructionmatrixlof
(5.23)calculatedfordiﬀerentnoiselevelsofthemembranesensorcells(0.25%,1%,2%,and5%of
themeanvalueofthemembraneintensities).Forfurthermodelparameterssee[252].Despitethe
pooropticalqualityofthepitorgan(centerimage),theoriginalinformationcanstillbeobtained
neuronallyfromthemembraneimage.

Thereconstructionperformanceoftheminimalmodel.Toillustratetheabilitiesofour
model,wepresentanexplicitexampleofhowpreciselythesnakecanreconstructitsIRsurroundings.
Weusethefamous1502drawingofAlbrechtD¨urerofahareconvertedinto8-bitgraylevelsata
resolutionof32×32;cf.leftpartofFig.5.10.Inthenextstep,wecalculatetheheatintensity
distributionrionthemembraneusingthetransferfunctionhkvgivenby(5.14).Wehavechosen
conservativebutrealisticvaluesthatcombinearelativelywideorganaperture(radius0.4mm)and
amoderatenumberofheatreceptors[252].Thecalculatedmembraneheatdistributionisshownat
thecenterofFig.5.10.
Theresultsofapplyingthereconstructionalgorithmtothemembraneimageareshownin
Fig.5.10.Thequalityimprovementisspectacular,providedthereisnottoomuchdetectornoise.
Themostimportantresultisthattheinformationneededtoreconstructtheheatpanoramaisstill
presentalthoughinvisibletooureyeinthemembraneimage.Thepresentmodelhasafairlyhigh
inputnoisetolerance.Forinputnoiselevelsupto50%,thehareisrecognizable.Sensitivityto
measurementerrorsislarger.Inourcalculations,onepixelofthereconstructedimagecorresponds
toabout3◦.Fordetectornoiselevelsuptoabout1%ofthemembraneheatintensity,agood
reconstructionispossible,meaningthattheedgeoftheharemaybedeterminedwithaboutone
pixelaccuracy.Atdetectornoiselevelsbeyondabout1%,theimageisnotsoeasilyrecognizable.
Choosingahigherσcancompensatethemembranenoiseuntilacertainnoise-levelrevealingat
leastthepresenceofanobject;cf.Fig.5.10.Thedrawbackofchoosinghighσvaluesisthatdetails,
viz.,variationsoftheheatimageare“treated”asnoiseandwillbesuppressed.Inaddition,in
thecaseofasimpleandclearinput–say,asmallwarmanimalinfrontofacoldbackground–
reconstructionmaystillbepossibleathigherdetectornoiselevelsaswewillseelater.

Predictedneuronalcharacteristics.Ourmodelisnotonlyabletopredictthesensoryperfor-
manceofthesnake’sIRsystembutcanalsoshedlightontotheneuronaldecodingmechanismused
tosharpenthemembraneimage.Theneuronalimplementationofthemodelisfairlystraightfor-

InfraredSystemofSnakes

Figure5.11:Predictedreceptiveﬁelds.Asexplainedinthemaintext,thestrengthoftheindividual
connectionscorrespondstotheentrylji.Theneuronsinthetopologicalmapreceiveexcitatory
(positive,red)aswellasinhibitory(negative,blue)inputfromthemembranereceptors.Thespatial
distributionoftheseexcitatoryandinhibitoryregionsonthemembraneiscalledthereceptiveﬁeld
ofacorrespondingmapneuron.Aswecanseeinpanel(A−E)thereceptiveﬁeldforasingle
mapneuronconsistsofaexcitatoryringlocatedattheedgesoftheheatproﬁlesurroundedbyan
inhibitoryring.Thisconstructionconstitutesanedgedetector.Wecanshrinktheradiusrofthe
organaperture(B:0.4mm,C:0.2mm,D:0.1mm,andE:0.05mm)untilwereceiveapin-hole-like
geometrythatiscomparabletothesetupinopticalsystemsemployingalens.Inbothsensory
systems,theIRsenseandthevisualsystem,weﬁndthesamemechanismsthatfollowthesame
mathematicalprinciplesofoptimality.Reconstructionparameterσ=3,distancebetweenaperture
andobjectis1m,anddistancebetweenapertureandmembraneis0.8mm.

ward.Thereconstructiontensorljicorrespondstoanetworkofsynapticconnectionsbetweenthe
membraneIRdetectorsandneuronsbuildingamapin,forinstance,theoptictectum[109,210,211].
Thestrengthoftheindividualconnectionswouldjustbethevalueofthecorrespondingentrylji,
basedonaratecoding.Theneuronsinthetopologicalmapreceiveexcitatory(positive)aswell
asinhibitory(negative)inputfromthemembranereceptors.Thespatialdistributionofthese
excitatoryandinhibitoryregionsonthemembraneiscalledthereceptiveﬁeldofacorresponding
mapneuron;cf.Chap.1.ItcanbecalculatedbyourmodelasillustratedinFig.5.11.Ring-like
structuresarisethat“detect”theedgesofprojectedimagescorrespondingtothepositionthatthe
mapneuronencodes.Asimilarsystem(calledlateralinhibition)isfoundinmammalianneurons
receivinginputfromtheretina;cf.Chap.2.Althoughtheexcitation-inhibitionpatternthere
isdisc-shaped(“small”aperture)ratherthanring-like(largeaperture)theexcitation-inhibition
patternnaturallyfollowsthesameprincipleofmathematicaloptimalityasthereceptiveﬁeldwe
havederivedhere.

InfraredSystemofSnakes

Insummary,thesnakedoesneedaccuratedetectorstogetareasonablereconstructionquality.
Indeed,theexperimentallydeterminedmembraneprecisionishigh.Thepitmembraneresponds
totemperaturechangesdowntoabout1mK[33,63].Aswediscussedinpart5.3,theresponse
ofthedetectors(FNEs)totemperaturechangesismuchmoreprominentandthusreliablethan
tostatictemperaturevalues.Wenowincludethisﬁndingandadapttheminimalmodelderived
aboveaccordinglybymeansofincorporatingthedynamicdetectorresponses.

delmoextendedThe5.4.2HereweextendtheminimalmodelwithrespecttothemajorﬁndingofSec.5.3concerningthe
neuronalcharacteristicsofsensoryencoding.Themajorresultisthattemperaturechangescould
beencodedneuronallymorereliably,i.e.,withamuchhighertemporalprecision,andmoreclearly
thanthestaticmembranetemperature(5.19).Wenextdiscussthisﬁndinginthelightofneuronal
decodingandadaptourminimalmodelderivedaboveaccordingly.

Detectorcharacteristicsinthelightofneuronaldecoding.Theminimalmodelclearly
informationdemonstratescanthatbethedecodedinformationneuronally.transmittedThetominimalthemopitdel,memhowbraneever,isfacesstillatherebigdraandwbacthatk.theIt
needshighlyaccuratesteady-statetemperaturemeasurementsonthemembraneandtheliterature
wlaceksreferanetoxpthelanationﬁndingsofofwhereSec.the5.2.1necessarywhereweprecisioncouldseecouldthatoriginatethetempfrom.eratureTogivconesometrastnisuminbtheers,
rangeofmKwithintheﬁrst100msofastimulusonset.Inaddition,themeasurementerrorsneed
is,tobtheelessFNEsthanhav10%etotodetectreconstructabsoluteaheattemppanoramaeraturecasonwetrastshaveofseabenout1previously;mKupcf.toF20ig.mK5.10.withThatat
leastsampling0.1inmKtervalsaccuracyto.encoGivdeenthearelevnoiselessanttempdetectioneratureprovcess,alues.theLookingdetectorsatthewouldneuronalthenrespneedonses200
(seefoundFig.inthe5.8)toeascendingncodenerthevesranofgetheofpitΔξme=m20braneK;ev(Fig.enif5.8)weweconsiderﬁndathefrequencytemporalrangeofpatternsayof20Hzthe
thecorrespsensoryondingsystemspikeandtrainsalmaosthighunbelievaccuracyableinincaserepresenofatingnoisythestimdetectionulusproseemscess.veryWecthereforehallengingnoforw
focusontheotherneuronalresponse,thedynamicansweroftheTNMstotemperaturechanges.

Theextensionoftheminimalmodel.Wecanthinkoftwodiﬀerentoriginsoftemperature
changesofthemembrane.Theﬁrstpossibilityisthesudden(dis)appearanceofaheat-emitting
toobjecmotvemencausingts.Wthearmriseoborjectsfallinofthethememexternalbraneworldtemptranslatingerature.Therelativsecondelytoandthepimosttmemnaturalbraneisleaddue
toobjectemptmovorallyes,ctemphangingeraturetempchangeseraturesappateartheatthesensoryedgescofellstheontheresultingpitmemtempbrane.eratureIfpsucroﬁlehaonwarmthe
membrane.Inencodingtemperaturechanges,theFNEsdonothingbuteﬀectivelyﬁlteringthe
edgesWeofcanthepitcalculatememanbraneinedge-ﬁlteredtensityversiondistributionfi(seeri.Fig.5.12)ofthemembraneintensitydistribution
rjbymeansoflinearﬁlteralgorithmssuchastheLaplaceﬁltermatrixtij[91],
fi=tijrj=tij(hjvξv)=hˆivξv.(5.24)
vˆTheheatabconovetrastξvequationontoandeﬁnesedge-ﬁlteredsimilarlytointensit(5.14)yaproﬁletransferfionfuncthetionmemhibrane.thatnoSincewmapsthetheexedge-ﬁlteredternal
imageapproximatesthetemporalderivativeoftheintensityproﬁleonthemembrane,wehave
essenextendedtialpropourmoertiesdelofofthethesnakdetectionemecinfraredhanism.senseinUsingathesimplesamebutmethoreasonabledologywaasythatdescribnoedwiinncludes(5.20)

InfraredSystemofSnakes

–(5.23),wecancalculatecorrespondinglyto(5.23)anoptimalreconstructiontensorlˆijusingthe
newtransferfunctionhˆivof(5.24)
lˆikhˆjvhˆkv(1+τ2)+σ2δjk=hˆji.(5.25)
Arrivingattheaboveequationwehaveimplementedaveryplausiblesecondstepofsensory
processing(seeChap.1),i.e.,wehaveincludedthepitmembranedetectorcharacteristics.Even
thisseemstobeasmallstep–aswehaveleftoutthederivationstepsanalogueto(5.20)–(5.23)
fortheminimalmodel–itresultsinabigadvantageaswewillseeinaminute.

Figure5.12:TheLaplaceﬁlter.Asexplainedinthemaintext,weusetheLaplacianﬁlterthat
can“detect”theedgesofthemembraneintensitydistributiontoextendtheminimalmodel.The
Laplacianﬁltermechanismworksinthefollowingway:(B)Theﬁlterkernelsequentiallytakesevery
pDepoinetofndingtheontheoriginalLaplaceimage,ﬁlterscaleskernelit,atandhand,subtractsitcantheaﬀectweighthetedareavofalue3×of3,itsasinneightheboringﬁgurep5oin×5ts.,
ormorepixelscenteredaroundtheevaluatedpoint.(C)Tousethismethodinthecontextofthe
minimalmodelwehavetoiwritetheﬁlterkernelaccordinglytotheindexingioftheoriginalimage
inimagetothe(A)linesontoofatheLapmatrixtlacianjforﬁlteredeachimageimage(pBoin)bt.yThemeansofLaplaceonematrixmatrixtmjnowultiplication.mapstheoriginal

advAdvanantagestagesthatofthearisefromextendedusingmothedel.dynamicTheinsteadextendedofmothedelstaticfeaturesresponse.thefolloFirst,wingthechangeconceptualin
neuronalactivityΔA≈80Hzofthedynamicresponsecausedbyanincreasingtemperature
contrastξisabout16dyntimeslargerthanΔAsta≈5Hzofthestaticresponsecausedbyanincreasing
tempemployerature16timescontrasmoret;cf.Secsampling.5.3.poinDuetstotothisdecofact,detheusingsignal.theThatdynamicmeans,responsetheallodetewsctiontheacsnakcureacyto
es.vimproΩ,theSecond,relativnoteconlyhangetheofabsolutefrequencychangecomparedoftoneuronaltheactivitactualyΔAfrequencyoftheoffreethenervedetector,endingsisbutincreasedalso
(5.19).Third,Thatacis,hangingthequalitIRyenandvironmenreliabilitt–ymoofvingtheobneurjectsonal–rdoespesonsenothevenindermorebutthanactuallyimproenableves.the
detectionprocess.IncaseofusingtheabsolutetemperaturecontrastξoftheIRenvironment
(minimalmodel),ξhastobestatic.Thatis,incaseoftheminimalmodel,movementsaretreated
asnoise.Incontrast,theextendedmodelisbasedonthedynamicresponse,i.e.,detectorresponses
tothechangessignaltoofthethemoIRendel.Thisvironment,decreasesdξ/dt.thTheus,noisemovlevelemenintsthearenotsignalrelatedmappingtoprnoiseocessbut.Ofconstitutecourse,
thereisalsoatemporal“integrationwindow”duringwhichasignalhastobeconstant.Sincethe

InfraredSystemofSnakes

dynamicresponse,i.e.,thesudden(about100ms)increaseinfrequencyoftheneuronal-detector
activity,istemporallyquitepreciseandfast,theintegrationwindowisnarrow,andthusthe
precise.aredetectorsFourth,weremembertwoﬁndings.First,inSec.5.2.1wehavederivedthatthetemperature
contrastξonthemembranehasnotreacheditssteady-statevaluewithintherelevantreaction
timeofthesnake.Thesignalξcanthusneverbeconstantandself-averaging,respectively.Second,
thereconstructionmechanismneedsveryaccuratesensorydata.Asignalthatisconstantduringa
certainperiodoftimecanbedeterminedmorepreciselybymeansofmakingmoremeasurements
andaveragingtheresults.Inthecaseoftheminimalmodel,ξisnotconstantandthuscannotbe
improvedbystatisticalmethods.Onthecontrary,thetimederivativeofthetemperaturecontrast
dξ/dtisalmoststaticwithintherelevanttimeframe.Astaticphysicalsignalprovidesaveraging
and√thusreducingadditivedetectornoise.Wecanatleastimprovethemeasurementbyafactor
∼twithmoremeasurements.Thatis,theaccuracyofhowwecandetectandencodethesignal
ishigherincaseoftheextendedmodel.
Insummary,wehavederivedanextensionoftheminimalmodelthatconnectssmoothlytoa
plausibleunderstandingofthedynamicresponsepatternfoundintheFNEs;seeFig.5.8.Toclarify
theoverallsensoryperformanceoftheextendedmodel,wenexthavetoquantifyitsdependency
uponnoise.SincetheunderlyingmathematicsoftheextendedisakintoaLaplacian-ﬁlteredversion
ofthemembraneimageonecannotexpectagoodnoiserobustnessbeforehand.Thisfactcould
cancelouttheaboveadvantages.Inthefollowingsection,wethereforecomparetheabilityofthe
twomodelsathandtoreconstructtheIRenvironmentandtoextractrelevantfeaturessuchasthe
objectpositioninvarioussettings.

extractioneatureF5.4.3AfterhavingseenabovethattheIR-sensorysystemofsnakescanreconstructtheheatenvironment,
wobejecnotswandaddresswhichthequanquestiontitiesofinﬂuencehowthepreciselysensorysnakpes,thaterformance.isourtwomodels,canlocalizewarm

Thenumerical“experiment”setup.Inthefollowingnumericalexperimentswepresenta
warmobjectwithinanon-uniformandnon-zerotemperaturebackgroundtoanartiﬁcialpitorgan.
WeassumeanIRenvironmentwith25◦Cmeantemperature.AswederivedinSec.5.2.1and5.3,
theexternalheatcontrastξextandthecorrespondingtimederivativedξext/dt,respectively,are
theappropriatestimuliforthepitorgan.WeaddaGaussian-distributedtemperaturebackground
contrastwithΓ◦CmeanandΓ◦Cstandarddeviationtostudytheeﬀectsofexternalnoiseand
signalcontrast.ThewarmobjectshowsaGaussian-shapedtemperatureproﬁlewithmaximalheat
contrastξmax(inrespectto25◦C)andstandarddeviation6.25cmthatislocatedatx=(x,y)
withx,y∈[−1m,1m],1minfrontofthepitorgan;cf.Fig.5.13.Foreveryrunwerandomly
positiontheobjectcenteratx,addanoisybackground,mapthewholeinfraredsceneontothepit
membraneandreconstructtheenvironment.Weusesimpleshapesbecausewenowcanextractthe
estimatedpositionxˆ=(ˆx,yˆ)easilybymeansoftakingtheaverageofthe50%warmestlocations
ofthereconstructedenvironmentandcomparetheestimatedpositionxˆtotherealpositionx;
cf.Fig.5.14.Duringthenumericalexperimentswemonitorthedependenceofthelocalization
performanceuponthebackgroundcharacteristics,namely,Γandthedetectornoiseχ;seeFig.5.15.
WehavetonotethatthefocusofthisworkisontheIRsensorysystemandnotonthemost
cognitivealgorithmforobjectrecognitiongivensensorydata,i.e.,apost-processingstep.Wehave
alreadyshown[251]howtoestimateboth,positionandshapeofobjects,givensensorydataofa
naturalsensorysystem.So,ourﬁndingspresentabottom-lineperformanceofthesnake’ssensory
abilitiesthatcanbebetterthanderivedhere–intheaspectofobjectrecognition.

InfraredSystemofSnakes

Figure5.13:Setupforlocalizationperformanceexperiments.(A)Wesimplifythewarmobjecttobe
atwo-dimensionalGaussian-shapedproﬁlewithstandarddeviation6.25cm,maximalamplitude
ofξmax=5◦C,andpositionx,andaddaGaussianrandombackgroundtemperaturecontrast
distributionwithmeanandstandarddeviationofΓ◦C.(B)ThisparameterizedIRenvironment
ismapped(minimalmodel:hikorextendedmodel:hˆik)ontothepitmembrane,anddetector
noiseχiisadded;cf.(5.15).Nowitdependsonwhichmoidelweconsider.Eithertheintensity
distributionriandthecorrespondingreconstructiontensorlk(minimalmodel)ortheedgesofthe
intensitydistributionfiandlˆik(extendedmodel)areusedtoreconstructtheIRenvironment.In
thelattercasetheedge-ﬁlteredmembraneimagefiapproximatesthetime-derivativefi≈dri/dt
duetoobjectmovements;seeSec.5.4.2.(C)Intheﬁnalstep,weextracttheestimatedpositionxˆ
ofthewarmobjectfromthereconstructedIRimageandcompareittotherealpositionxduringn
independentruns:seeFig.5.14.Toquantifyhowtheparametersaﬀectthesensoryperformance,
wepresenttheroot-mean-squareerrorbetweenx=(x,y)andxˆ=(ˆx,yˆ)inFig.5.15.

Sensoryperformance“experiments”.Withinthesetupoutlinedinthelastparagraph,we
varythesignal-to-nosie(SNR)ratiooftheexternalheatcontrastξbymeansofincreasingΓgiven
aconstantξmax=5K.Inaddition,westudyhowthedetectornoiseχionthepitmembraneas
givenby(5.15)aﬀectssensoryperformance.Theresults(Fig.5.14)illustratehowpreciselythe
positioncanbeestimatedfromtheIRenvironment.Incaseofmoderatenoiseboththeminimal
model(leftpanelsinFig.5.14)andtheextendedmodel(rightpanelsinFig.5.14)canreconstruct
thepositionofthewarmobjectquiteprecisely.
Inthissection,thestandarddeviationofthemembranenoiseσχisrelatedtothemaximal
valueofeithertheintensitydistribution(minimalmodel:σχ∼rmax)ortheedgesoftheintensity
distribution(extendedmodel:σχ∼fmax)onthemembrane.Thepreviousdeﬁnitionofmembrane
ofthemeanvalueofthemembraneintensities:σχ∼1/niri;seeFig.5.10.Sinceahigher
noisepresentsthestandarddeviationσχoftheadditiveGaussiannnoisevariableχiinpercentages
temperaturebackground–viz.,increasingΓ–alsoincreasesthemeanvalueofintensitiesrion
membrane,thetwodiﬀerentkindsofnoisewouldnotbeindependentincaseoftheprevious
deﬁnition.Soweredeﬁnethenoiseleveltobenowrelatedtothemaximalvalueofthemembrane
intensityriortheedgesofthemembraneintensityfi.Thisre-deﬁnitionalmostcompletelydecouples
.ΓandχFurthermore,weseeinFig.5.14thatinbothmodelsthecapabilitytolocalizeawarmobjectis
stillavailablebeyondthemembranenoiselevelof1%butdecreasesfast.Independentlyofeach
otherboththeexternaltemperaturebackgroundΓandthemembranedetectornoiselevelsχ,play
animportantroleinpredictingthesensoryperformanceandinevaluatingexperimentalresults.
Surprisingly,theestimatedpositionsxˆdonotonlybecomemorevaguebutalsoclusteredaround
themiddleofthepitapertureduetoincreasingnoiselevels.Thelattereﬀectismoreprominentin

InfraredSystemofSnakes

respcaseofonsesthetomitempnimaleraturemodel;ccf.hangesleftinpanelsprincipleinFig.access5.14.sensoryThedataextendedwithmohigherdelemploaccuracyyingasthecompareddynamic
totheminimalmodel.Onthecontrary,sinceLaplacianﬁlterstypicallyenhanceimagenoise[91]
wasecouldcomparedexpetoctthethatmintheimalextendedmodel.moIfdelthisthatisisthebasedcase,onthealessLaplaciannoiseﬁltertoleranistlessdecodingtoleranttoalgorithmnoise
wouldcancelouttheadvantagesgainedthroughamoreaccuratesensoryencodingmechanism.At
ﬁrstglance,theextendedmodel,however,isasrobusttonoiseastheminimalmodel;seeFig.5.14.

Figure5.14:Localizationperformanceexperiments.Thetwocollectionsofninepanelsillustratethe
eﬀectsofthetwoparametersχi,themembranenoise(x-axis)andΓ,parameterizingtheexternal
IRenvironment(y-axis)forboththeminimalmodel(left)andtheextendedmodel(right).Ineach
ofthe18panelswehavedepictedrealpositionsx(panelx-axis)versusestimatedpositionsxˆ(panel
y-axis)of2500trailscorrespondingtothesettingdescribedinFig.5.13.Weseeclearlythatboth
modelscanreconstructthepositionoftheinfraredobjectverypreciselyincaseofmoderatenoise
levels.Inbothmodels,localizingawarmobjectispossiblebeyondthemembranenoiselevelχ
of1%butdecreasesfast.Furthermore,thelevelofcontrastΓwithintheexternaltemperature
backgroundplaysanimportantroleinpredictingthesensoryperformanceasthelocalizationgets
worsewithincreasingΓ(seey-axis).Interestingly,notonlydoestheestimationaccuracydecrease
butforhighΓtheestimatedpositionstendtobemoreclusteredaroundthecenteraswell.Above
all,theextendedmodelrobustnesstonoiseseemsatleasttobecomparablewiththeminimalmodel
–itisevenbetterintheparameterrangedepictedabove;butseealsoFig.5.15.

Tomathematicallyquantifytherobustnesswithrespecttobothtemperaturebackgroundand
detectornoise–theperformanceofthetwomodels–wedeﬁnetheroot-mean-squareerrorErms
betweenrealxandestimatedcoordinatexˆ
Erms=(x−xˆ)2/n(5.26)
nthroughrobustnessnasindepcomparedendenttoruns.theInextendedFig.5.15model.weseeThethatnoisethetoleranceminimalofthmoedelextendedshowsmoadel,higherhowevnoiseer,
isgoodenoughtoallow,incombinationwiththeadvantagesofthesensoryencodingstep,i.e.,the
higherqualityoftheneuronalresponse(5.19),theconclusionthattheextendedmodelisthemodel
e.oichcofConsequently,weagainposethequestionofwhatisthestimulustotheinfraredsystemsincewe
havederivedthesuperiorityoftheextendedmodelemployingthedynamicresponse.InSec.5.2.1

InfraredSystemofSnakes

wehavederivedthatthestimulushastobetheexternalheatcontrastξextaccordingtothe
thermodynamicsofthesysteminsteadoftheexternalabsolutetemperature.Theﬁndingsof
sensoryencodingonthelevelneuronalcompartmentsinSec.5.3havethenhintedtowardsthetime
derivativeoftheexternalheatcontrastdξext/dttobetheadvantageousandappropriateinputfor
thepitorganinsteadofξextortheabsolutetemperature.Thenumericalperformanceexperiments
havenowunderlinedthisassumption.Althoughintheliteraturetheabsoluteexternaltemperature
isreferredtobeingthestimulustothesensorysystemwecanalsoﬁndexperimentalstatements
thatsnakesrespondbesttomovinginsteadofstationarywarmobjects[94].Aswehavediscussed
above,thelatterstatementscorrespondtodξext/dtandhenceprovideanexperimentalevidence
forourtheoreticallyderivedﬁnding.

Figure5.15:Quantiﬁcationofthemodels’localizationperformance.TheplotsshowErmsforthe
andminimalχ(y(left)-axis)andusingtheanextendedoptimalmoσ.delFor(righeacht)insetdofepnoiseendenceupparametersontheΓnoiseandχ,wparametersehaveΓv(xaried-axis)σ
The(2500minimalrunsformoeacdehlcomemploysbination)alargertoﬁndparametertheoptimalspaceinσthatwhichlominimizescalizationtheofloawarmcalizationobjecterrorisErquitems.
wgooelldpenoughossible,toviz.,notincancelwhichoutErmstheisadvsmall.antagesTheofnoiseusingthetolerancedynamicoftherespextendedonsesucmohasdelistheathigherleast
qualityofencodingatemperaturestimulus.

Insummary,wehaveshownmathematicallythattheIRsignalsoriginatingfromanexternalheat
distributionandgettingprojectedontothepitmembranestillsuﬃcetoreconstructboth,animage
oftheexternalheatdistributionandthepositionofwarmobjects.Furthermore,wehaveclearly
demonstratedthatincludingcharacteristicsofthedetectionprocessintothereconstructionmodel,
viz.,focussingonthedynamicresponse,enablesareasonableunderstandingoftheremarkable
performanceoftheinfraredsystemofsnakes.

InfraredSystemofSnakes

5.5DiscussionandOutlook
Inthischapterwehaveanalyzedtheinfraredsystemofsnakesbystudyingthethreestepsof
sensoryprocessingthatwehadintroducedinSec.1.1.First,usingtheeyeglassesofphysicswe
havelookedatthemembraneheatingprocessandcorrespondingexperimentalﬁndingsofthepit
organssoastoidentifytherelevantequationsdescribingtheﬁrststepofsensoryprocessing,viz.,
thesignalmapping.Indoingso,wehavequantiﬁedhowtheheatingprocessdependsonexternal
quantities–suchasdistanceortemperaturecontrastofawarmobject,anatomicaldata,and
time.Second,wehavepresentedanionicmodelthatcombinesrecentanatomicalﬁndingsand
presentmodelsofheatreceptors.Thatis,bytakingtheeﬀectsoftransientresponseprotein(TRP)
channelsandtime-dependentsub-thresholdcurrentsintoaccountwewereabletoreproducethe
neuronalresponsesofthefreenerveendings(FNEs)notonlytostatictemperaturesbutalsoto
temperaturechanges.Ourﬁndingsofsensoryencoding,furthermorehinttowardssynchronization
eﬀectsoftheFNEsthatcouldberesponsiblefortheoutstandingtemperaturesensitivityofthis
sensoryarchitecture.Third,forthelaststepofsensoryprocessing,i.e.,neuronaldecoding,wehave
providedbothaminimalandanextendedmodelthatareabletoreconstructtheexternalheat
distribution,i.e.,theinfraredimageoftheenvironment,givenonlytheheavilyblurredmembrane
image.Acomparisonofbothmodelswiththeirresponsetodetectorandexternalnoisehasrevealed
theabilitiesandlimitationsofthisIRsense.Ourﬁndingsbothﬁtwellintotoday’spictureand
extendtheunderstandingoftheinvolvedprocesses.
Moreover,wehaveposedandansweredthequestionofwhatisthetruestimulustothis
sensorysystem.Intheﬁrstsectionofthischapterwephysicallyderivedthatitisnottheabsolute
temptemperatureerature.butIntheaddition,temptheeraturecequationsontrasthavofetheshownexternalthatwtheorldtempthateratureeﬀectivconelytrastaﬀectsonthethememmembranebrane
doesnotreachitssteady-statevalueduringtherelevantresponsetime.Sincethetimederivativeof
thetemperaturecontrastonthemembranestaysalmostconstant,wehavestartedtoconsiderthe
wetimehavederivfounativdeofmoretheevidenceexternaltoheatsupportconthistrasttothesis.betheThestimsupuluseriortoqualittheyIRofthesystem.sensoryInthisrespconsehapterto
temperaturechangesascomparedtostatictemperaturesisoneofthem.Totesttheassertion,we
haveextendedtheminimalansatzmodelingthereconstructionperformance.Insummary,wehave
thenotbtempeenoralabletoderivﬁndativeorofderivtheeexternalreasonableheatconargumentrasttsistheagainstbutappropriateonlysintimfavulusorforforthethesnakclaimes’thatIR
system.Withinthiswork,notonlyarenewinsightspresentedbutalsonewquestionsforfurther
researchhavearisen.Inthecontextofsignalmapping,experimentallygainedvaluesoftheinvolved
quantitiessuchasthebloodﬂowkinematicsortheheatconductanceofthemembranetissue
areneeded.Systematicandquantitativemeasurementsofthewarmingprocessdynamicswould
allowforunderstandingthefunctionsoftheindividualphenomenasuchasbloodﬂowandsurface
structureone-by-oneandininterplaywithoneanother.
dynamicsOnthearelevelneededoftosensoryfurtherencoenlighdingtenmothedelsquestionincorpoforatingwhatexpmakeserimenthetalpitmemdataofbraneTRPsocsensitivhannele.
ApromisingissuecouldbesynchronizationeﬀectsoftheFNEs.
Inthecontextofreconstructionmodelsweseetwofutureapproaches.Ontheonehand,
theoreticalmodelsshouldbecomemorebiological.Thatis,theyshouldbebasedonneuronaldata
–ifpresent–todescribesensoryperformanceonthelevelofspikingneurons.Theextendedmodel
basedoncharacteristicsofthesensoryencodingprocess,i.e.,thedynamicresponse,providesagood
startingpoint.Ontheotherhand,nobodyhaseveraddressedthequestionofhowtheaccuracyof
theIRsenseandthusthetopographyofthecorrespondingneuronalsynapticconnections,arises.
Thereisaclearlackofexperimentaldataonthisissue.Fromthetheoreticalpointofview,onecan
imagineinstructivelearningasfoundintheauditorypathway,orsomekindofauto-focusalgorithm

InfraredSystemofSnakes

thatusesimagestochasticstoimproveitself,tocausethedevelopmentofthissensorysystem.In
thenextchapterwewillstudyamodel,calledintegratedMultimodalTeaching,thatcouldbea
reasonableexplanationofhowtheinfraredmapandthevisualmapintheoptictectumgetaligned.
Theﬁndingsderivedinthischapterarenotpurelyofbiologicalinterest.Asourmathematical
modelsshowagoodperformancewithoutemployingcomplexcalculations,wehavealreadystarted
tosetupaprojecttoimplementourﬁndingsintohardware;cf.Fig.5.16.Theoverallgoals
aretoevaluateourmodelsinrealsettings,toisolateindividualaspectsofthemodelsandthe
involvedphysicsthatarenotpossibletostudyinabiologicalcontext,andtodevelopandtest
learningalgorithms.Onthehardwaresidethegoalsaretoclarifythepossibilityofimplementing
bio-inspiredcognitivecircuitsinsilicontoimproveopticalIRinstruments.Firststepsarepromising,
andwearelookingforwardtothisinterdisciplinary,bionicapproach.

Figure5.16:Bionicexperimentsetup.Asdescribedinthemaintext,wearegoingtosetupan
artiﬁcialpitorganexperimentincooperationwithErikJungandAlexanderHilgarthfromthe
FraunhoferIZM,Berlin.Theinputprojectormapsthesignalontothedetectorsoftheartiﬁcialpit
organ.Werealizeafullyﬂexiblemotor-controlledgeometry(ﬂexiblepositionofthepitorganwith
respecttotheinput,variableorganaperture,adjustabledistancebetweenapertureanddetector)
toevaluatethepredictionswehavederivedinthiswork.Moreover,wewilldevelopandimplement
“reality-feedbacked”learningalgorithmsandminiaturizethissetupintheprogressoftheproject.

Aﬁnalremarkregardingthesensoryperformanceoftheinfraredsystemofsnakes:Itseemsthat
notonebigdevelopmentbutratherthecomplexinterplayofseveralspeciﬁcstepssuchasavery
thinandisolatedmembrane,adensecapillarynetwork,FNEsemployingtemperaturesensibleionic
channelsthatareorganizedtoacompactnetworkcalledterminalnervemass,neuronaldecoding
mechanisms,etc.haveledtooneofthemostspecializedandremarkablesensorysystemsfoundin
nature.Wewereabletoanswersomebutnotallofthequestionsconnectedwiththesesteps.
Inthenextchapter,wefocusonthegeneralquestionofhowaprecisealignmentofdiﬀerent
sensorymapscandevelopintrinsically.Thealignmentofdiﬀerentmapsisfundamentalfor
multimodalintegrationthatisalsofoundinthesnake’soptictectum.There,theinfraredmapand
thevisualmapinteractfaithfully,forinstance,bymeansofperforming“AND”(e.g.visibleand
warm)and“OR”(e.g.visiblebutnotwarm)operations[109,210,211].

Millionssawtheapplefall,butNewtonwasthe
onewhoaskedwhy.
hBarucesMannBernard

6.IntegratedMultimodalTeaching

Inthischapterwecomebacktothetopicoflearningandaddressglobalstructuresunderlying
theevolvementofsensorysystemsandtheirpreciseinteraction.Weintroducetheconcept
ofintegratedMultimodalTeaching(iMT)thatisbasedonsupervisedspike-timing-dependent
plasticity(sSTDP)andmultimodalintegration.Weshowthatamultimodalteachercanensure
properintrinsicmapformationandalignment.Thedominanceofvisioninguidingmapalignment
isdemonstratedtobeanaturalconsequenceoftheexceptionalprecisionofthevisualsystem.

ductionIntro6.1Ourabilitytoactinadynamicandcomplexenvironmentreliesﬁrstandforemostontheprecise
interactionofthediﬀerentsensorymodalitiesthatformacongruentperceptionoftheexternal
world.Thispreciseinteractionofthediﬀerentmodalitiesisnotpresentatbirthbuthastobe
learnedduringtheﬁrstperiodofsensoryexperienceinlife[271,280].

Figure6.1:Thebasicideaofintegratedmultimodalteaching(iMT).Allmodalitiesareintegrated
aintoteacanhinginsignaltegratedm(blackultimoarrodalws)mapthat(graleadsyarrotoanws).Theadaptationactivitofyofthetheunimomultimodaldalmaps.mapWethenexplaininducesthe
detailedmodelsetupemployedinthepresentchapterinSec.6.2.2.

Theprocessofsensorycalibrationiscommonlyseenasvision-guided;cf.King’sreview[148].
Severalremarkableexperimentsshowthatdestructionordisturbanceofthevisualpathwayleads
todisorganizedandabnormalsensorymapsinnon-visualmodalities.Theseﬁndings,obtained
insupporthamsterthe[200idea],ofcat[vision278,as280teac],clahingwedmofrogdalit[y.44],Howferretever,[150],vision-guibarndeowld[map159],andcalibrationinsnakisesnot[96as],
evidentasitseemsatﬁrstglance.First,sensorymaps,althoughimprecise,canevolvewithoutany
visualdespiteainput[missing146,158visual,271].system,Exptheerimentsauditoryonandvisuallydeprivsomatosensoryedanimalssystemsandcanhumansdevelophaveshonormally;wnthatsee
[152,231,234]fordetails.Second,behavioral[30,68,77,249,258]andphysiological[202]studies

IntegratedMultimodalTeaching

haWveallaceshownandthatStein[vision277,can279]halsoavebepoininﬂutedencoutedbythatotherreceptivmoedalities.ﬁeldsinThird,thesupanderiormostimpcolliculusortan(SC)tly,
thatalreadyallmoshrinkdalitiesatshotimeswawheretempornoallyvisualcoincidingneuronsdevareelopmenpresentt.inFwhichurthermore,evenvisiontheyhavimproeves.observButed
ifhowvisioncanitshouldimprovacteasitselfteac?Thingogether,modalitthey,abhoowvecanparitadoguidexicalcaexplibrationerimentalwithoutandbeingtheoreticalpresenﬁntdingsand
questionthecurrentpictureofvisionasthepredominantguidingmodality.Thatis,whatisa
coherentalternativetovisionastheteachingmodality?
moIndalitiesthiscbeginshapter,atwtheetaktimeetofwoappﬁndingsearanceasofmstartinultimogpdaloinints.tegrationFirst,[the279].formationSecond,ofthediﬀerenmotor,t
hasi.e.,bameenultimosuggesteddalmaptoinprothevideSCaasteacwhellingasinsignalitsfornon-mammalianauditoryspacehomolog,alignmenthet[optic180];cf.tectumFig.(OT),1.4
forSCconnectivity.Therefore,wesuggestanintegratedMultimodalTeachingconcept(iMT);
seeﬁndingsFig.as6.1.toWhoewexplaindiﬀerentthemwoell-prodalitiesvencanbedominanalignedceofwiththerespvisualecttosystemeachandother.unifyIncondoingtradiso,ctorwey
inontroaduceconceptualaninlevtegratedel.Tmoultimothisdaend,lteacweherfocusandondiscussmapitsinformationtrinsiccwithinharacteristicstheSC/OTandbyimplicationsmeansof
long-termpotentiation(LTP)andlong-termdepression(LTD)ofsynapsesthatcanbedescribedby
movsSTDPe.theirWeleasensoryveoutorgansthe(eyquestiones,ears,ofwhiskdynamicers,etc.)adaptationindepasendenapptlyeariofngtheirinbmanodyy.Theanimalsmecthathanismcan
underlyingthisadaptationisnotknown,butretinotopiccoordinatesystemsseemtoplayakey
horolewev[83er,].mAttemptsultisensorytoandmodelthispredominanissuetlycanmonbeosreviewensoryedlayiners[82are,99m,240utually,241].alignedWithintogaintheaSC/OT,uniﬁed
multisensoryrepresentationofsensoryspace[147,261].Thecombinedsensoryinformationcan
Inthenotherbewusedords,toatgeneratethisstagedirectionalallsensoryresponsesmapsinsharetheaSC/OTcommonmotorrepresenmap[tation166,167system.,179,Sim216il,ar260to].
othercalibrationtheoreticalappearingmodelsin[the78,con287],texttheofthesuggesteverydiMTcommonconceptsensoryconcenspace.tratesInonadditionthetoquestionderivingofmapthe
generalconceptforiMT,weapplyourmodeltoconcreteexamples.Usingthismodel,wereanalyze
ofinWdetailallacetheandconcreteStein[277exp,er279ime]nintsoftheKnlightudsenofouretal.iMT[159]concept.andcompareFinally,wthemewithconcludethebyﬁndingsgiving
experimentallytestablepredictionsofourapproach.

6.2IntegratedMultimodalTeaching(iMT)
Ourconceptofintrinsicsensorymapcalibrationstructurallyconsistsofastackofunimodal
maps,eachconnectedwiththemultimodalmap(cf.Fig.6.1).InouriMTconcept,diﬀerent
unimounisensorydalmapsmocdalitiesancurrenareﬁrsttlybemergedfoundintowithinthemtheultimoSC/OT,dalmap.whereSucthishamsituationultimoisdalexpinerimentegrationtallyof
wellunimodaldescribmaps.ed[261Th,us,279].theInmtheultimonextdalstep,maptheinducesinategratedteacmhingultimosignaldalthatmapproguidesjectsthebackcalibrationtothe
does(formationnotoriginatandealignmenfromat)singleprocessmoofdalitythe,fordiﬀerenexample,tunimovisiondal,butmaps.fromtheTherefore,integratedtheteacmultiherminodalput
map.InForgeneral,adetailedmapcalibrationunderstandingcanofbeourdividedmodelintodesign,twopleasediﬀerentseeSec.classes.6.2.2.Theﬁrstdescribesthe
situationinwhichsensorymapshavetobelearnedfromscratch.Connectionsbetweensensory
systems(inputmaps)andrepresentingmapscontainnoinformationandarerandomlydistributed.
Thesynapses;inconnectionshamathematicalvetoevtermsolveduringdescribedthebySTDPcalibration.WeprodenotecessbythismecaseansasofLTPformationandofLTDtheofmathpse.
Thesecondclassrequiresacollectionofalreadypre-wiredmaps[271]andcorrespondsto

eachingTdalMultimoIntegrated

anstimulusalignmentpositionofismapheres,dforisplacedinstance,induringcomparisongrotowththeorpinositionshiftingoftheexpteacerimenhertsinput.[158,By159].meansTheof
activity-basedsynapticplasticity,thesynapticpatternofthemapismodiﬁedsoastocompensate
forthemisalignmentbetweenthemapandteacherinput.
TocarefullyanalyzetheconsequencesofapplyinganiMTsignalwehavetounderstandmap
proformationcess(asandshowninalignmenFig.tf6.2)romaproconstitutesceduralapclosedointoffeedbacview.kloToopthisthatend,consistsweofnotethreethatreptheeatingiMT
ansteps.inFirst,tegratedmduringultimomultimodaldalmap.integrSecond,ation,duringdiﬀerentmultimosensorydaltemoachingdalities,theareinoptimallytegratedmultimergedmoindalto
mapalignmeninducest,ofanthediﬀereninhibitorytteacsensoryhingmaps.signalThird,thatguidesduringtheunimodalcalibrationmapproadaptioncess,,i.e.theformationadaptingandand
adaptedunimodalmapsagainmodifytheintegratedmultimodalteacher.
Thethreephaseswehavedescribedaboveareinherentlyconnectedbymeansofthreemajor
questions:1)HowdounisensorymapsdetermineiMT?2)HowdothecharacteristicsofiMT
together,inﬂuencethesemapthreeadaptation?questionsandpro3)videHoawdostructureesiMTforourcalibrateiMTdiﬀeconceptrentandunimwilloadaldmaps?ditionallyTakguideen
usthroughtheanalysisoftheiMTprocessinthefollowing.

Figure6.2:MainproceduralaspectsofiMT.Combiningdiﬀerentunimodalmapsintoamultimodal
mapleadstocertaincharacteristics.Whatarethesecharacteristicsandhowdotheyforman
instructiveteachingsignalforthemapcalibrationprocessthatcanbedescribedbymeansof
supervisedspiketiming-dependentplasticity(sSTDP)?Toclosethefeedbackloop,wediscusshow
diﬀerentunimodalmapsadapttoacommonteachingsignal.

6.2.1HowdounisensorymapsdetermineiMT?
aTosensoryanswermothedalitﬁrstymasaitsjorneurquestion,onalwerepreseninitiallytationhavonetoaclarifyspatialthemap;essenseetialSec.terms.1.2.WOnerefersuchtoa
maponecaninterprettheﬁringrateasthelikelihoodtoﬁndastimulusatthepositionthatthe
neuronsencode.Inthismanner,sensorymapsrepresentpositionestimatorsorpopulationcodes
[60,116,131,227].Dependingonthestatisticsoftheﬁringproﬁle,wecanattributeameanµ
correspondingtotheestimatedpositionandastandarddeviationσdescribingtheestimator’s
accuracytoeachsensoryestimator.Theintegrationofdiﬀerentsensorymapscanthereforebeseen
tobeequivalenttoacombinationofpositionestimators[2,67,102,113,126,165,202].
Althoughtheexactneuronalmechanismastohowdiﬀerentmodalitiesareintegratedintoa
multimodalmap[60,61]isstilllargelyterraincognito,thefamilyofBayesiancombinationschemes
mathematicallydescribestheresultofoptimalmultimodalintegration.Forthesakeofsimplicity,
wewillfocusonthemaximumlikelihoodcombinationofonlytwomodalitiesinthefollowingtext,

IntegratedMultimodalTeaching

Thenamelyvisual,visionand(V)auditorandymapsauditionare(cA).Theharacterizedcasebofythreetheirmomeans,i.e.dalitiesisthepdescribositionedinestimates,AppendixµVC.2.for
visionandµAaudition.Furthermore,theaccuracyofthepositionestimatesisgivenbyσV(vision)
andσA(audition),andillustratedbyFig.6.3.

forFigureaudition,6.3:Multimovision,dalandinthemtegrationultimooftdalwomapunisensoryassumedasmaps.TheGaussianﬁringfunctionsproﬁleswithofmeanneuronalµA/Vmaps/M
inandtegratedstandardmapisdeviationassumedσtoA/Vb/eMincantegratedbeinbymterpretedeansofasapmositionaximumlikestimatorselihoodofthemethodstimurepresenlus.Theted
byand(6.1)isthandusthe(6.2).most(A)Theprecisemofultimoalldalmaps.map(Bsho)IfwsthethesmalleunisensoryststandardmapsaredeviationshiftedσµMV<=σVµA,<σtheA
multimodalmapµM(redproﬁle)islocatedmorecloselytothemoreprecisemodality(blueproﬁle)
.(6.3)(see

ThemultimodalpositionestimateµMasamaximumlikelihoodcombinationfromcoinciding
visualandauditoryinputsisgivenby[135,144]
22σσµM=σA2+VσV2µA+σA2+AσV2µV(6.1)
inwithtegration,thestandardthemultimodeviatidalonσMestimatorofthemfeaturesultimothedalpsmallestositionpesossibletimate.Asstandardaconsequencedeviationofσgivoptimalen
MybσσVAσM=σA2+σV2.(6.2)
Foradetailedderivationofthetwoaboveequations,pleaseseeAppendixC.1.
pWositionsecanµalsobycomputemeansofthe(6.1)distancetobetweenthemultimodalpositionµMandtheunimodal
A/V|µM−µA/V|=σ2A/V|µA−µV|/σA2+σV2.(6.3)

dalMultimoIntegratedeachingT

Thatis,themultimodalandthereforetheteachingsignalislocatedmorecloselytothepositionof
themoreprecisemodality,e.g.,visionincomparisontoaudition,asshownhere.
Theaboveequations(6.1)to(6.3)illustratehowwecandescribeanintegratedmultimodal
mapataspeciﬁcpositioninspaceandtime.Nextwehavetotakeintoaccountthattheintegrated
multimodalmapwillprovidetheguidingsignalforunimodalmapformationandalignment.The
equationsofoptimalcombination,however,donotdescribehowunimodalmapschangeduringthe
learningprocess.Mapformationandalignmentareaneﬀectofsynapticplasticity(mathematically
describedbySTDP),aswewilldiscussinSec.6.2.2.Inthissectionwefocusonfundamental
characteristicsofthemultimodalmapthatfollowdirectlyfromoptimalmultimodalintegration.
Fornow,wepresumethattheunimodalmapsthatformthemultimodalmaparevariableintime.
Moreprecisely,ontheonehand,thepositionoftheunimodalmapsµA/V(t)canandwillvary
duringthecalibrationprocess.Ontheotherhand,themapscanimproveinprecisionsuchthat
σA/V(t)decreasesaswell.ThesetwoeﬀectsleadtoavariableiMTwithtime-dependantµM(t)
andσM(t)duringthecalibrationprocessoftheunimodalmaps.
Inshort,themultimodalmapispositionallyﬂexibleoverthecourseofthealignmentprocess
becauseitisthecombinationoftwoormoresensorymaps(6.1)to(6.3)thatthemselveschange
duringthelearningprocess.Wewillnowdiscussingreaterdetailwhatpositionalﬂexibilitymeans.
Wi.e.,ehathevecmeanµharacterizedofitsﬁringeachproﬁle.mapinThthisus,wpapeerﬁrstbycitshangestandσard(Fig.6.4deviationA)andσandseethatestimaatorsensoryposition,map
Vthatbecomesmoreprecise(blueproﬁle)“attracts”thepositionofthemultimodalmap,i.e.,the
learningsignal.SensorymapsthatincreaseinaccuracygetmorestablepositionallyduringsSTDP
learningprocesses.Second,ashiftingsensorymap,forinstance,vision(blueproﬁle)inFig.6.4B,
inducesashiftofthemultisensorymaptowardsthestationarysensorymap(yellowproﬁle).The
upshotofiMTevenwithoutincludingeﬀectsofsSTDPinthecontextofmapalignmentisalready
thatamorestaticsensorymapattractstheteachingsignalandwillthereforehardlyadaptatall.
Insummary,wehaveanalyzedcharacteristicsoftheoptimallyintegratedmultimodalmapand
havetherebyshownthreeessentialaspects.First,themultimodalmapisalwaysmoreprecisethan
anyunimodalmap;cf.(6.2).Second,theoptimalmultisensoryinstructivesignalislocatedmore
closelytothepositionofthemoreprecisemodality;see(6.3).Third,onthebasisofanadaption
ofthemonosensorymapsthroughSTDPlearningrules,(6.1)and(6.2)describehowposition
andprecisionofthemultimodalteachervariesforgivenµA/V(t)andσA/V(t)oftheinputmaps.
Equippedwiththesecharacteristicsoftheteachersignal,wenowanalyzehowthecharacteristicsof
iMTinﬂuenceunimodalmapadaptationonthelevelofsynapticplasticity,thatis,inphysiological
andmathematicaltermsforsSTDPmapalignment.
6.2.2HowdoiMTcharacteristicsinﬂuencemapadaptation?
Intheprevioussectionwecouldseethatanintegratedmultimodalteacherhasaspeciﬁcvariance
andpositionthatbothdependonthecharacteristicsoftheinputmaps;cf.(6.1)and(6.2).In
addition,thesecharacteristicswillvaryduringmapcalibration.Inthissection,wethereforediscuss
howteacherswithdiﬀerentaccuraciesandteachersthatareshiftingduringthelearningphase
t.alignmenmapaﬀectTheresultsaspresentedinthefollowingarebasedonnumericalsimulationsofaninhibition-
mediatedSTDPmapalignmentprocess.Thebasicstructureofourmodelconsistsofthethree
moteacherdalitiesmoadalituyditionTviaA,STDPsomatosensorylearningSrules;,andseevisionFig.V6.5.thatTheadaptfundamentothetalmprinciplesultimodalofinhibition-inhibitory
mediatedSTDPmapalignmenthavebeendescribedelaboratelybyFriedelandvanHemmen[78].
Allsensorymapsareone-dimensionalmapsofNPoissonneurons[145].Eachmodalityconsistsof
aninputmapwithaGaussianﬁringproﬁle
fiA/S/V(y):=SA/S/Vexp−(y−xi)2/2σ2A/S/V(6.4)


IntegratedMultimodalTeaching

Figure6.4:Characteristicsofanintegratedmultimodalmap.DuringtheiMTlearningprocessthe
diﬀerentmodalities(µA/Vchanges)alignwiththeinstructivesignalprovidedbythemultimodal
map,andtheirreceptiveﬁeldsbecomemoreprecise(σA/Vdecreases).Giventhatthemultimodal
mapisacombinationoftheunimodalsensorymaps,itsprecisionandpositionaredeterminedby
thesensorymapsandconsequentlychangeaswell.Therefore,wevaryonemodality(blueproﬁle)
inprecision(A)andposition(B)andleavetheothermodality(yellowproﬁle)unchanged.The
resultingdevelopmentofthemultimodalteacher(redproﬁle)isdepictedintheuppergraphs.(A)
Themodality(blueproﬁle)thatgetsmorepreciseattracts(6.1)andsharpens(6.2)themultimodal
teacher(redproﬁle).(B)Inaddition,themodality(blueproﬁle)thatalignswiththeteacher(red
proﬁle),inducesthemultimodalmaptoshiftaswell.Thedistancebetweenthemultimodalteacher
andthestationarymap(yellowproﬁle)thereforechangesaswell(cf.(6.3)).

for0≤i≤N,withstimuluspositiony,thei-thmapneuron’spreferredpositionxi,standard
deviationσA/S/V,andSA/S/Vbeingthemaximalamplitudeoftheactivity.Fortwodiﬀerentmaps,
e.g.audition(A)andvision(V),theaccuraciesdiﬀerandσA=σV.Theinputlayerprojectsvia
all-to-allsynapsesJjA/S/V(t)ontoneuronjofthemaplayer.Inaddition,eachneuronofthemap
layerreceivesinhibitoryinputfromonecorrespondingneuronoftheteachermodalityT.The
synapsesconnectingteacherandmaparemodeledasstaticone-to-onesynapticconnections.
artiﬁcialWeinteacterruptherthatthemisnotultimodalgeneratedfeedbacfromklotheopmultimocompareddaltorepresenFig.6.1tationandbut6.2shobywssimtheessulatingentialan
characteristicsthatwederivedinSec.6.2.1.Aneuronpoftheteachermapthereforerespondstoa
stimulusatpositionyinformofan“inverted”Gaussianfunction
fpT(y):=ST1−exp−(y−xpT)2/2σT2(6.5)
withxpbeingthepreferredpositionofneuronp,STbeingthemaximalﬁringactivityoftheteacher

eachingTdalMultimoIntegrated

Figure6.5:Modelarchitecture.Amultimodalobjectsuchasawaterdrophittingtheskinproduces
sensoryinputforaudition,somatosensory,andvision.WemodeltheinputbymeansofaGaussian
fiA/S/V(y)soastostimulatejneuronioftheinputlayer.Neuronsoftheinputlayerprojectthrough
all-to-allsynapticconnectionsJ(t)toneuronsjofthemaplayer.Thesesynapticconnections
willadapttoaligntheoutputofA/Sthe/VmaplayertotheinhibitoryteacherinputbymeansofsSTDP.
HereweusethismodelsetuptostudytheconsequencesofaniMT,whichisthemultimodal
representationofallsensorymodalities.

neurons,andσTbeingthewidthoftheteacherinﬂuence.Theprocesswehavejustdescribed
mathematically,whereateachersuppressestheactivityinthemaplayersatlocationsdiﬀerent
fromtherealobject’spositiony,iscalledselectivedisinhibition[78,127].
Toanalyzethealignmentprocessweperformnumericalexperiments.Weintroduceanartiﬁcial
shiftΔinthesensorypositionofthemodality,e.g.,realizedthroughprismsintheKnudsen
[158,159]experimentonthebarnowl.Inmathematicalterms,wereplaceybyy+Δin(6.5).
Thesensorymodalities,say,auditionA,somatosensoryS,andvisionV,subsequentlychangetheir
synapticconnectionsbetweentheinputandthemaplayertocompensate(follow)theshift;see
6.5.Fig.Thesituationinournumericalexperimentscorrespondstothesituationofshiftingvision
bylenses[158,159].AsexplainedinSec.6.3.1,ashiftofvisioninducesadisplacementofthe
multimodalteacher.Consequently,ashiftofallsensorymapswithrespecttothepositionof
thecombinedmultimodalteacheroccurs,thatis,againstintuition,lensesshiftingvisionhardly
displacevisionwithrespecttotheteacherbutdisplaceallothersensorymapswithrespecttoour
multimodalteacher.Therefore,thebiologicalexperimentsandournumericalexperimentsagree.
Thechangeswithinthesynapticconnectivitypatternduringthealignmentprocesscanbe
describedandexplainedbySTDP[78,145].Toquantifychangeswithinthesynapticpatternwe
introducethemapestimationerrorEasaleast-squareerror.Itmeasureshowwellamapcan
reproducepositionalinformationofmsystematicallyvariedincomingsignals,

m
E=xi−xiT
=0i

2/m.(6.6)81

IntegratedMultimodalTeaching

Here,xidenotesthemeanpositionoftheﬁringproﬁleofthemap,andxiTrepresentsthetheoretical
position.ThesmallerthevaluefortheerrorE,thebetterthemapreproducesthereal(theoretical)
positionofanobject.TheparametersusedinournumericalexperimentsaregivenintheTableC.1.
Multimodalteacherwithvaryingvariance
Oneoftheessentialcharacteristicsofthemultimodalteacherisitsabilitytoimproveinprecision
duringthealignmentprocessasaconsequenceofdynamicunimodalmaps;see(6.2).Afterbirth,
mostanimalsﬁrsthavetolearnhowtointerpretthesensoryinformationtheyreceive.Allavailable
mapsasgivenbygeneticsareveryinaccurate[181,271].Furthermore,WallaceandStein[279]
observedthatmultisensoryneuronsincatsyoungerthan28daysreliablyrespondtoinputsfrom
morethanasinglemodalitybutdonotshowmultimodalenhancementordepression.These
observations,however,aremeasurementsonthelevelofsinglecells.Subsequentconclusionsinthe
contextofneuronalmapsandestimatorsareelusive.Therefore,itisfundamentallynecessaryto
showthatlearningstillfunctionsoratleaststartswithaninaccurateteacher.Beforewestudy
theeﬀectofadynamicteacheronsSTDPmapalignmentwewillclarifyhowteacheraccuracy
isdeﬁned.Wenotethatthelessaccuratetheunimodalmaps,thelargerthevarianceofthe
multimodalteacher(6.2),thatis,theestimatedpositionandthustheteachingsignalisspatially
lessaccurate.Consequently,wemodeltheteacheraccuracybyaddingaGaussiandistributed
randomshiftΔTtothecorrectteacherposition.Asaresultofthenumericalexperimentswitha
varyingteachershiftΔT,weseethattheunimodalmapsadaptmoreslowlyandwithalargermap
estimationerrortoalessaccurateteachersignal;cf.Fig.6.6.Withinagiventimewindow,an
inaccurateteacherthereforeslowsdownbutdoesnotpreventthemapalignmentprocess.Wecan
thereforeconcludethattheiMTconceptdoesnotdependcruciallyonaconcreterealizationofthe
tegration.indalultimom

Figure6.6:InﬂuenceofteacherprecisionΔTonmapalignment.Wedepictthemapestimationerror
E(6.6)formapalignmentwithamultimodalteachervaryinginprecision.Themapestimation
errorbetweenteacherpositionandrepresentedpositiononthemapdropsfasterandtoalower
levelwithamoreprecise(lowerΔT)teacher.Theprecisionoftheteacherthereforeinﬂuencesthe
temporaldevelopmentofmapalignment.

Multimodalteacherwithdynamicallychangingposition
AsillustratedinFig.6.4,theintegratedmultimodalteacherpositionshiftsduringthealignment
process.Therefore,wemodelteacheractivitiesthatcontinuouslyshiftinspaceandanalyzehow
theirshiftingvelocitiesinﬂuencemapalignment.Forthetimestept=2000s,Fig.6.7showsthe
temporaldevelopmentofsensorymaps,i.e.theirsynapticpattern,modiﬁedbymeansofteaching

IntegratedeachingTdalMultimo

signalsshiftingwithdiﬀerentvelocities.Wenotethatthedynamicsoftheteacheressentially
determinestowhichdegreeamapcanadapttotheteacher.Theslowertheteacherpositionshifts,
thebetteramapcanadapttotheteacher.Moreover,thecriticalvelocityuptowhichmapadaption
worksproperlydependsontheprecisionofthesensorymodalitythatisreﬂectedintheprecisionof
theinputmap.Alessprecisemodalitycanstilladapttoateacherwithagivenvelocity,whereasa
precisemapcannotfollowanymore;cf.Fig.6.7,changefromtoptobottom.

perfFigureorm6.7:edwithInﬂuenceinputofdynstandardamicallycdeviationhangingσ=teac0.03herp(topositionsline),σon=map0.05(balignmenottom)t.Simandteaculationsher
standarddeviationσT=0.02.Fordetailsregardingtheparameters,seeAppendixC.3.Weshift
theteachersignal,thatis,theinverseGaussianproﬁle,inspace.Theredlinesindicatethemean
positionoftheteacher.Inournumericalexperimentstheteacherpositionstartsshiftingfromthe
simdashedulatedredtimeline(ofΔ=20000.1s.).TheThedistancestraightbredetwlineeentheindicatestworedhowlinesfarthedepteacendsheronhastheshifteddiﬀerentwithinshiftingthe
vtheelocitdiagoyofnal.theTheteacboherxatsignal.thetopTheleftshosynapticwsthepatternsynapticofthepatternsensorycausedmapsbyareasloinitiallywlyloshiftingcatedteacaroundher.
toWiththearighcertaint,thedelateacy,herthevelosynapticcitypatterndoubles.ofAstheamapconsequehasnce,adaptedthetomaptheadaptsteachermucmap.hlessByunsteptilpingthe
velocityreachesacriticallevel(toprightbox)wherethemapdoesnotadaptatall.Thevelocity
oftheteacherthereforeessentiallydeterminestowhichdegreeamapadaptstotheteacher.This
eﬀectcanbecompensatedbydecreasingmapprecisionasrealizedfromthetoptothebottomline.
Wmap.ethenTheseecriticalthatvtheelocitysynapticofpatternnon-adaptationofalessisthereforeaccuratemaphigherstillforaloadaptsw-pvreerycisionwellmaptothethanteacforhera
map.high-precision

meansWeofconcludesSTDP.thatHowevdiﬀerener,ttheteacteacherheraccuraciesaccuracywillnotinﬂuencespreventhettempalignmenoraltdevofdiﬀerenelopmenttmapsand,btoy
somedegree,theaccuracyoftheﬁnalstate.Wehaveseenthattheshiftingvelocityoftheteacher
pﬂexibleositionthandeterminesanaccuratetowhicmhapbdegreeecauseaitmapcancanadaptadapttoatothfasteretedyacher.namicAallesslycaccuratehangingteacmapher.ismoreThe
precisionofthediﬀerentcontributingunimodalmapsthereforeseemstodeterminehowtheyadapt
totheteachersignal.Inthenextsectionwewillstudytheseeﬀectsindetail.

IntegratedMultimodalTeaching

6.2.3HowdoesiMTcalibratediﬀerentunimodalmaps?
Havinganalyzedﬁrstthecombinationofanintegratedmultimodalmapasitarisesfromunimodal
andmapslastandstepthe(seeeﬀectFig.ofa6.2)howdynamicallythecessentialhangingcteacharacteristicsheronofmapthealignmendiﬀerentt,wmoesimdalities,ulateinintheparticular,third
thediﬀerentaccuraciesofthesensorysystems,inﬂuencemapformationandalignment.
ResultsofoursimulationsofmapformationandalignmentareshowninFig.6.8.Basedon
thesenumericalexperimentsweobservethatmapformationandalignmentbymeansoftheiMTis,
inofmapgeneral,pformationossibleandforinputsalignmenoftdepdiﬀerenendtontheaccuracies.accuracyHowevofer,thethesensoryqualityandsystem.temInporalpassing,progresswe
notethatwithinthealignmentprocess,ahigh-precisionmapreachesamodiﬁedsynapticpattern
thatisagainmoreprecisethanthatofalow-precisionmap.Moreimportantly,ahigh-precision
mapadaptsmoreslowlytotheteacherinputthanamapwithlow-precision.Inconclusion,the
highertheprecisionofasensorymap,themorestaticthemap.Wewillseeinthenextsectionhow
theseﬁndingscanexplainexperimentsofsensorymapformationandadaptation.

plotsFiguresho6.8:wtheInﬂuencemapofestimationvaryingerrorinputEasmapgivenprecisbyion(6.6)(σ)fromonmaunimopdalformationmapsinandcomparisonalignment.toThethe
teacher.Themapsvaryinprecision(inputstandarddeviationσ)asindicated.Thegraphsshow
isthemapenlargedtoestimationclearlyerrorshowEforwhichmapmapformationadapts(A)faster.andforBothmapplotsshoalignmenwtthat(B).mapsThebwithoxedahigherregion
precisionadaptmoreslowlytotheteacherinputthanlower-precisionmaps,butregainhigher
precisionafterthemapcalibration.

6.3Modelresultsandexperimentalﬁndings
AfterdetailedanalysisandbeingequippedwithadeeperunderstandingoftheiMTconcept,we
returntotheexperimentspresentedintheintroductionandreviewtheminthelightofouriMT
concept.Indoingso,wedistinguishbetweenthepreviouslymentionedexperimentsthatcanbe
explainedbyavision-guidedconceptofmapcalibrationandthosethatcannot.TheiMTconcept
wepresenthereoﬀersexplanationsforboth.
Inthefollowingwewillanalyzeagroupofexperimentsthatmotivatedtheideaofvisionas
thedominantguidingsensorymodality:theprismglassexperimentsofKnudsenandcoworkers
[127,158].Wereviewtheremainingexperimentswhiledenyingavision-guidedmapalignment.We
thenillustrateusingexperimentalevidenceasexamplesfromSteinetal.[277,279]howaniMT
conceptcouldwork.Finally,wemakesuggestionsandtestablepredictionsfornewexperiments
concept.iMTtheonbased

eachingTdalMultimoIntegrated

AnimalσAσVµµVA−−µµMAµµVV−−µµAM
Human1◦0.02◦99.96%0.04%
Barnowl2◦0.3◦97.8%2.2%
Cat8◦0.2◦99.93%0.06%
Table6.1:Visualandauditorylocalizationcapabilitiesofhuman[57,194],barnowl[106,157],and
cancat[b19e,42].distinguishedThevisualinoneresolutionsdegreeofaretheoriginallyvisualﬁeld.givenWeinhavcyclesepassumederdegree,thei.e.,resolutionshowmantobyelinesthe
inverseofthesevalues.Onthebasisofthesevarianceswecombinethetwounimodalmapstoa
multimodalteacherandcalculatethedistanceofeachmaptothemultimodalmapdueto(6.3)in
comparisontothedistancebetweenthetwomodalities.

6.3.1Experimentsprovision-guidedmapformation
Themostprominentexperimentssupportingvision-guidedmapformationaretheshiftingexper-
imentsbyKnudsenetal.[127,158].Here,prismsshiftthevisualsystemofowlsbyacertain
angle.Afteralearningperiodtheneuronalprojectionsfromtheauditorymaparerearrangedto
compensateforthemisalignmentbetweenvisualandauditorymaps;seeSec.1.3intheintroduction.
Thevisualmap,incontrast,remainsconstantduringthewholealignmentprocess.
CanouriMTconceptreproducethedominanceofvisionwithinthelearningprocess?Toanswer
thisteacherquestionmapaswegivhaenvebysim(6.3)ulated.Wenotealignmenthattofthemapsvisualwithsystemdiﬀerentusuallyvarianceshasaanmducdhistancehigherstospatialthe
resolution(aboutafewarcminutes)thanallothersensorysystems,e.g.theauditorysystem
(aboutafewdegrees);seeTable6.1fordetails.Wenowrevisitsynapticplasticityduringaprism
shiftexperimentfromthepointofviewofiMT.

•Initialstate(Fig.6.9,top)
Auditoryandvisualmapsareshiftedwithrespecttoeachotherbecauseofanartiﬁcialshift
ofthevisualsystemthatisintroducedbyprisms.Giventhatthemultimodalteachermap
isacombinationoftheunimodalmaps,itsactivitymeanhasshiftedaswell.Anadaptation
processofallunimodalmapsisinduced.Againstnormalintuition,shiftingvisiondoesnot
onlyresultinashiftbetweenvisionandthemultimodalteachingsignal;itinsteadresultsina
shiftofallsensorymapscomparedtothemultimodalteachermap.Asaconsequenceofthe
greaterprecisionofthevisualsystem,themultimodalteachermapismainlydeterminedby
thecharacteristicsofthevisualmap,thatisσML≈σVandµML≈µV;cf.(6.1)and(6.2)).
Equation(6.3)neatlyillustratesthattheshiftbetweenvisionasthemostprecisesensorysystem
andthusthemultimodalteachermapissmallerthanforanyothersystem.
•Intermediatestate(Fig.6.9,middle)
Synapticconnectionstotheauditorymapstartchangingtocompensatefortheshiftbetween
auditionandthemultimodalteacher.BasedonourpreviousstudiesaspresentedinSec.6.2.2
and6.2.3,weknowthatbecauseofitshigherprecision,thevisualmapadaptsmuchmoreslowly
thantheauditorymap.Theauditorymapthereforestartsadaptingtothemultimodalteacher.
Theteacherasacombinationofthetwomodalitiesthusshiftstowardthevisualmap.The
auditorymapadaptsagaintotheteacherinputandhencecontinuestoshifttowardthevisual
map,whichstaysalmoststatic;seeFig.6.4,B.
•Finalstate(Fig.6.9,bottom)
Thetwomapshavebeenrealigned.Thepositionofthevisualmaphashardlychanged,whereas

IntegratedMultimodalTeaching

Figure6.9:Numericalrealignmentexperiments.Weillustratetheshiftingprocessfortheauditory
map(A)withvarianceσA=0.07andthevisualmap(V)withσV=0.03.Thedistancesbetween
teacTheherarrayandplotsunimoshodalwthemappsynapticositionsareconnectivitchosenypatternaccordinglyfortovision(6.3)(leftforthecolumn)correspandondingauditionvariances.(right
mapcolumn).A,theThevisualsynaptmapicV,strengthsandthehereteacbyherranmgeapbetareweensho0wnandin1.theThemiddleactivitycolumn.proﬁlesTheofthecolorcoauditorydeis
speciﬁedinthebottomrightcorner.Theplotillustrateshowsynapticplasticitydependsonthe
faraccuracylessthanofathemolessdality.Inaccurateconcreteauditoryterms,maptheA.moreprecisevisualmapVshiftsmoreslowlyand

eachingTdalMultimoIntegrated

theauditorymaphasshiftedalmostthewholedistanceinducedbytheprismsatthebeginning
t.erimenexptheofKnInudsensumandmary,othersour[iMT127,158concept].Hocanwever,reproducesimilarthenumericaldiﬀerentstepsresultsofcanthebeshiftingobtainedexpforerimenanytstwofo
systemarbitraryonlysensoryreﬂectsmapsthewithmoregeneraldiﬀeringvconceptariancesof.aTherefore,dominancetheofacobsecurrvacyedfollodominancewingofnaturallythefvisualrom
concept.iMTthe

6.3.2Experimentscontravision-guidedmapformation
Intheintroductionofthischapterwereviewedsomeexperimentsthatquestionavision-guided
ofitslearninghighprocess.precisionThebutanalysisthatofvisionourisnotconceptstatic.hasshoItwncanbthatemovisiondiﬁedplabysyaadyndominanamticallyrolecbhangingecause
multimodalteachermapandthereforebyanyunimodalmap[68,77,202,249,258].
Furthermore,theiMTconceptshouldbeabletoreproduceformationofunimodalmapsin
lovisuallycalizesoudeprivndasedpreciselyanimals.asStudiesindividuals[152,231with,234]normalshowvision.thatblindMeasuremenhumans,tsofcats,WallaceandandferretsSteincan
[277,279]supportthecriticalviewofvision-dominatedlearningbyshowingthatreceptiveﬁelds
butalreadyisshrinkdominatedatbytimesvisionwheninnocasesvisualinwhichneuronsitisaarevailabpresenle.t.Learningispossiblewithoutvision
butInitthehascaonmuctexthoflowoureriMTprecision.conceptFaormationmultimoofdaltheteacotherhermapsmapiscanthereforeexiststillwithoutpaossiblevisualwithinmap,
certainlimits;cf.Fig.6.6and[146,158,271].
animal,Inthetherebfolloywingstudyingtext,wewhetherwillsomreproduceatosensoryformationandofauditoryunimomdalapsmapscandevinealopvisuallywithoutdeprivvisioned,
i.e.,withaveryimpreciseteacher.
•InitialSomatosensorystate(Fig.and6.10,auditorytop)mapsareonlycoarselyprewired,andthevisualmapdoesnot
existimpreciseatall.aswTheell;mseeultimoSec.dal6.2.2mapforasadetails.combinationofsomatosensoryandauditorymapsisvery
•Intermediatestate(Fig.6.10,middle)
Evwhileentheirwithvaveryariancesdecimpreciserease.teacThehermapearlierthesimauditoryulationsandillustratethatsomatosensorythelearningmapsstartprotcessowithimprovane
impreciseteacherstartsveryslowly,correspondingtothetopmostcurveinFig.6.6.Nevertheless,
twtheounimosensorydalmapsmapsasaimproveconsequence.slightly,asWithdoesthethemimproultimovingdalteacmapherthat(seeisFig.acom6.6),thebinationoflearningthe
accelerates.cesspro•Finalstate(Fig.6.10,bottom)
Thetwomapsarealignedandhavereachedtheirbest-possibleresolutions.
Together,multimodalteachingillustratesthattheiMTconceptisasuccessfulconceptfor
incalibratingparallel,diasﬀreperentortedunimoindalexpmaps.erimentsInsucourhinastrinsic[277,279].learningprocessallmapsdevelopandimprove

6.4DiscussionandOutlook
IntheapplicationpresenoftciMThaptertowmapefhaveormationintroandducedthealignmencontceptleadsoftointhreetegratedfundamenMultimotallydalnewTeacideas.hing.First,The

IntegratedMultimodalTeaching

Figure6.10:Numericalmapformationexperimentswithanimprovingteacher.Activityproﬁlesof
theauditorymapA(σA=0.03),thesomatosensorymapS(σS=0.045),andtheteachermapare
showninthemiddlecolumn.Theblurredteacherprojectsontothemapswithaprecisionthatstarts
atΔT=0.1andthenreduceslinearlyintimetoΔT=0.02fort=7500s.Theblackbarsbelow
theﬁring-activityplots(middlecolumn)denotetheregionswithintheteacherisprojectingforthe
correspondingtimestep.ThearrayplotsshowthesynapticconnectionpatternJAforaudition
(leftcolumn)andJSforthesomatosensorymodality(rightcolumn).Thesynapticstrengthsrange
between0and1.Theinitialsynapticpatternisonlycoarselyprewiredanditisadditionallyblurred
bymeansofrandomlydistributedsynapsestrengths.Thecolorcodeisspeciﬁedinthebottomright
corner.Despiteanimpreciseteacher,bothmapscandevelopproperly.

eachingTdalMultimoIntegrated

theintegratedteachercontainsallavailableinformationofthemapsinvolved.Second,theteacher
isintrinsictothesystembecauseitisgeneratedonlyfrominformationavailablewithinthesystem
ofmaps.Third,asaconsequence,theteacheritselfisdynamicwiththepotencytoshiftinposition
andtoimproveinprecisionduringthelearningprocess.
Moreover,thepresentedresultsprovidethebasicinsightthatthemorepreciseasensorysystem
is,themoreslowlyitadaptstoinstructivesignals.Thisnaturalconsequence,whichwedenoteas
dominanceofaccuracy,explainsobservationswhereadominanceofvisionwithinthealignment
processmaybewronglyinterpretedaspurelyvisuallydriven.
state-of-the-artthetoarisonCompHere,wehavesimulatedmultimodalteachercharacteristicsformapcalibrationonthelevelof
spikingneuronsandsynapticplasticity,namely,sSTDP.Theteachersignalinﬂuencestheunimodal
mapsbyselectivedisinhibitionasproposedbyanincreasingnumberofexperimentalﬁndings,
forinstance,intheICC-ICX-OTpathway[14];seealsoSec.1.3oftheintroduction.Taken
together,ouriMTconceptthereforeconnectstwofundamentalneurophysiologicalphenomena:
inhibition-mediatedmapcalibrationandmultimodalintegration.
Previoustheoreticalstudieshavefocusedononlypartsoftheseaspects.GelfandandMysore
[86,204]proposedmodelsforHebbianmapcalibrationbasedontheassumptionthatvisionserves
asteachersignaltomodify,forinstance,synapsesoftheauditoryprojectionpathway.Asalready
pointedoutintheintroductionofthischapter,visionasteachersignalneverthelesshastobe
questioned.SteinandWallace[277,279]thereforesuggestedaconnectionbetweenmultimodalmap
andmapformationincats.ThelatestexperimentalstudiesonbarnowlsfromBerganandKnudsen
[14]alsoshowthatcross-modaleﬀectsexistintheICXandmayinﬂuenceexperience-dependent
calibrationofconvergingrepresentations.
Includingtheseexperimentalﬁndings,Wittenetal.[287]developedatheoreticalmodelfor
bimodalmapalignment.Intheirmodel,thesynapsesofboththeauditoryandthevisualmap
areadaptedduetoHebbianlearning.Bythesemeans,themodelcanreproducethedominanceof
visionsuchthatthechannelwiththeweakerorbroaderreceptiveﬁeldsalwaysexhibitsmostorall
oftheplasticity.However,thetheoreticalmodelofWittenetal.isexclusivelybasedonexcitatory
inputsandratecodes.Thelatterassumptionsarebothquiteimplausibleinthecontextofmap
alignmentinICXandOT.Concretely,experimentaldatafromBerganandKnudsen[14]suggest
thatinstructivesignalstotheICXaregatedbyinhibition.Inaddition,Kempteretal.[145]
demonstratedthattheassumptionsunderlyingrate-basedHebbianlearningarenotnecessarily
valid,especiallyintheauditorysystemofthebarnowl.
modelofIncludingmapthealignmenlastmentbytionedmeansexpoferimenselectivtaleresults,disinhibitionFriedelonettheal.lev[78el]ofdevspikingelopedaneurons.theoreticalThe
modelshowsthataninhibitoryteacherisessentialtorealignalreadycalibratedunimodalmaps.
However,themodelonlycontainsplasticitywithintheauditorysystem,thatis,thevisualsystem
isstaticandprovidestheguidingsignalforauditoryspacealignment.
ThepresentiMTmodelcombinesthefundamentalaspectsofthemodelsfromWittenetal.
[287]andFriedeletal.[78]:spike-timing-dependentlearningbasedonbothselectivedisinhibition
andfullplasticitywithinallcontributingmodalities,thatis,allavailableinformationisintegrated
intoamultimodalmapasaﬁrststep.Inasecondstep,themultimodalmapprovidesaninstructive
signalformapcalibrationtoallmodalities.Therefore,theiMTmodelcanexplainthedominant
roleofvisioninmapalignment,plasticityinmodalitieswithoutvision,plasticitywithinthe
visualmodalityguidedbyothermodalities,plasticityofnon-visualmodalitiesguidedbyanother
non-visualmodality,andanycombinationofthesesettings.
TestablepredictionsoftheiMTconcept
GiventhatwehaveshownthatouriMTconceptcanreproducebothexperimentssupportingand
questioningvision-dominatedlearning,itisnowtimetooﬀersomesuggestionsregardinghowthe

IntegratedMultimodalTeaching

iMTconceptcanbeexperimentallyveriﬁed.Todistinguishvision-dominatedteachingfromiMT
wehavetobypassthedominanceofvision.Wecaneitherstudyartiﬁcialsituationsinwhichthe
precisionofsensorymapsisequal,investigateanimalswithapoorlydevelopedvisualsystem,or
observemapformationwhilemultimodalmapsareexperimentallydeactivated.
Toﬁrstequalizetheprecisionofsensorymaps,onecould,forinstance,performshiftingexperi-
mentswithprismsthatdisplaceandblurthevisualinputatthesametime.Second,studyingmap
formationandalignmentforanimalswithamoreprecisesensorysystemthanvisioncouldalso
identifyifthereisageneralprinciplesuchasdominanceofaccuracy.Athirdpossibilitywouldbe
todeactivatethemultimodalintegrationduringthelearningprocess.GiveniMT,deactivationof
theandmalignmenultimotdalshouldmap(s)failincorrespthisond(s)case.toInacondeactivtrast,ationforaoftheteacvision-dominatedhingmodalitteachy.ingMapscenario,formatione.g.
vision-guidedmapalignment,nonegativeeﬀectwouldbeobservedonthelearningprocess.
okOutlo

iMTFiguresetup6.11:shoOutlownoink:Fig.iMT6.5netbwyorkmeanssetupofwithincludingfeedbacktheconmultimonections.dalinOnecantegrationextendprocesstheoncurrenthet
levelpreviouslyofspikinginSec.6.2neurons.andThatdepictedis,weinFig.automatically6.2.First,incorptheporatetheost-synapticfeedbackcurrenmectshαanismgenerateddiscussedby
itheincomingspikesPfromtheneuronsiofthediﬀerentsensorymapsA/S/Varelogarithmically
summedup;Ij=ilogαi.Second,theprobabilitytoevokeaspikeinneuronjofthemultimodal
jmapofthecanmbeultimomodeleddaltomap.relateThird,expweonenusetiallyantointheterneuroncompletlayeerinputto“invcurrenert”tIthejinjectedexcitatoryintoactivitneurony
proﬁletoaninhibitorydisinhibitionsignalthat,fourth,guidesmapcalibrationofthesensorymaps;
sensorypathwayandinputisfadedascomparedtoFig.6.5.Inotherwords,withthenetwork
setupdescribedabove,wecanneuronallycalculateanoptimalmultimodalmapemployingallthe
characteristicswehaveusedfortheiMTinthischapterbymultiplyingtheactivityproﬁlesof
thesensorymapsonthelevelofspikingneurons;fordetailsseeAppendixC.Wehavealready
inimplemenprincipletedpandossibletestedbutthisthesetupfeedbacandkmecthehanresultsismloneedsokfurtherpromising.Mapclariﬁcation.formationandalignmentis

Insummary,wehaveintroducedthenewideaofanintrinsicmultimodallearningprocessand
insodoinghaverevealedinterestingnewaspectsofmultimodalmapformation.Thequestion
of(multimodal)mapformation,however,isnotyetfullyunderstood.Onthetheoreticalside,
futuremodelsshouldincludeatwo-wayfeedbackbetweenunimodalmapsandthemultimodal
teacher.Wehavealreadyimplementedandstartedtotestamoresophisticatedmodelonthelevel

eachingTdalMultimoIntegrated

ofspiking-neurons;seenetworksetupinFig.6.11.Inshort,spikesfromthesensorymapsA/Vare
summedlogarithmicallyprovidingtheinputcurrenttotheneuronsofthemultimodalmapMthat
aremodeledtorespondexponentiallytotheinputcurrent.Thiscorrespondstoamultiplication
ofspikesandmodelsoptimalmultimodalintegrationonalevelofspikingneurons;see(C.6)in
AppendixC.Theactivityofthemultimodalteacheristhen“inverted”viainter-neuronsand
projectedbacktothemapneuronsofthesensorysystemsA/Vtoprovideaninhibitoryteaching
signal.WearelookingforwardtoderiveanexplicitmodelofiMTincludingfullneuronalfeedback
dynamicswithoutartiﬁcialnormalizationterms.
Ontheexperimentalside,experimentsmaywelldeliverconvincingevidencefororagainst
aconnectionbetweenmultimodalprocessingandmapcalibration.Itwillbeinterestingtosee
whetherasystemofmapssuchaswhatisfoundintheSC/OT(seeFig.1.4)isabletocalibrate
intrinsically,andwhethermultimodalmapsplayakeyroleincalibration.

InpresentingtheconceptofiMT,wehavenotonlyaddressedthequestionofhowglobal
structurescanenablelocalmechanismssuchassynapticplasticitytoformcomplexbrainstructures,
e.g.,theSC/OT,wehavealsogivenanotherlineofevidencethatsensoryexperienceisthedriving
forceforbraindevelopment.Togiveaﬁnalstatementbeyondthischapter,thisisreallyremarkable
sincethebrainexiststoprocessthequantitywhichdeterminesitsfunctioning,viz.,sensory
experience.Thatiswhywebelieveitissoimportanttounderstandsensoryprocessing.

App

endices

A.Appendix:OptimalMapformation

A.1Arecipeformakingmaps
TobringtolifethemathematicalframeworkofSec.2.2,wepresentaneasystep-by-step“recipe”
toﬁndtheoptimalconnectivityinarealisticbiologicalsetup:
•First,wederivethetransferfunctionhix(t)thatdeterminestheresponseofthedetectoritoa
stimuluspulsethatoccurredttimeunitsagoatpositionx.
•Next,wecalculatetheFouriertransformHixofthetransferfunctionhix(t).
•Wechoosesuitablevaluesofτandσ.Ingeneralthenoise-to-signalratioτcanbeassumed
tobemuchsmallerthan1foranymeasurablesignal.Incontrast,σneedstobeestimatedin
dependenceuponthesituationathand[76,75,252].
•WethencalculatethematrixentriesMijasgivenbyEq.(2.19)andinvertthemodelmatrix
.M•WemultiplytheinvertedmatrixM−1bythevectorHixsoastoﬁndtheinputconnection
x.Lstrengthsi•Finally,wecalculatetheinverseFouriertransformofLixsoastoﬁndtheconnectionstrengths
lix(t).

A.2Nonlinearitiesininformationprocessing
Thepresentedmodelassumesalinearrelationbetweenstimulusanddetectorresponse.Fora
numberofsensorysystems,however,weﬁndnon-linearitiesinthemappingprocess.First,the
transferfunctionhcanbeanon-linearfunctionh˜.Second,theneuronaldetectorresponsecanbe
non-linear,typicallylogarithmic[169,213,172,136,53].Incaseofalogarithmicresponseanda
non-lineartransferfunctionthebiologicaldetectorresponser˜hastoberewrittenfrom(2.1)as
tr˜i(t)=logdxdτsx(τ)h˜ix(t−τ).(A.1)
spaceall−∞Toapplyourmodelweﬁrsthavetoincorporateanadditionalcomputationalstepcancelingthe
logarithm.Inabiologicalsystemthiscanberealized,e.g.,byneuronswithexponentialﬁring
behavior.Assumingsuchaneuronalstepr˜i(t)asin(A.1)reducesto(2.1)withanon-linearh˜.
Wecanlinearizeanon-lineartransferfunctionbyaredeﬁnitionofthesignals→s˜.Thatis,we
identifyappropriatecharacteristicsofthestimulusthatarelinearlyrelatedtor.Forexample,
insteadoflookingattheheatdistributionT(x,t)wecanconsidertheintensitydistributionofthe
correspondingradiation∼T4(x,t)duetotheStefan-BoltzmannLaw.Inthisway,areasonable
redeﬁnitionofdetectorresponseandsignalcanallowforanoptimallinearstimulusreconstruction.

A.Appendix:OptimalMapformation

A.3Remainingderivationstepsleadingto(2.16)
Inthefollowingweelaboratesomestepsskippedinthederivationof(2.16)inthemaintext.In
doingsowetakeadvantageofideasduetothecalclulusofvariations[138].Wethereforestartfrom
(2.6)asaconditiontominimizetheexpectationvalueofthequadraticerrorwithrespecttothe
optimalreversetransferfunctions,theconnectionstrengthsljx(t).Thisleadsto
∂[sx(t)−sˆx(t)]2
∂ljx(t)=0foreveryj.
⇔[sx(t)−sˆx(t)]∂sˆxx(t)=0.(A.2)
∂lj(t)
Inordertosolve(A.2),weexpandtheestimatesˆx(t)usingEqs.(2.3)and(2.4)fromthemaintext
givingsˆx=χilix+χiλix+dysyhiylix+syhiyλix+syηiylix
i+syηiyλix+ξyhiylix+ξyhiyλix+ξyηiylix+ξyηiyλix.(A.3)
Thevariationofsˆin(A.3)leadsto
∂∂lsˆx((tt))=χj+dysyhjy+syηjy+ξyhjy+ξyηjy(0).(A.4)
x
jAsbefore,weassumeontheonehandthatallnoisetermsaswellastheexpectationoftheinput
arestochasticallyindependentofeachother.Ontheotherhandweusethefactthatallnoise
termshavezeromeanandthesignalisself-averaging.Withthesetwoassumptions,theexpectation
valuesspartialˆs/∂landsˆ∂ˆs/∂lfrom(A.2)canbewritten
sx(t)∂sˆxx(t)=dysx(t)(syhjy)(0)(A.5)
∂lj(t)
andsˆx(t)∂∂lsˆx((tt))=(χilix)(t)χj(0)+dydy(syhiylix)(t)(syhjy)(0)
x
ji+(syηiylix)(t)(syηjy)(0)+(ξyhiylix)(t)(ξyhjy)(0)

+(ξyηiylix)(t)(ξyηjy)(0).(A.6)
Inthenextstepwesubstitutethecorrelationtermsasgivenin(2.10)ofthemaintext.Toillustrate
thecalculations,whichsimplify(A.5)and(A.6),weanalyzetwoisolatedtermsfrom(A.6)asan
example.Theothertermsaretreatedinasimilarway.Weﬁrstsimplify
dydy(syhiylix)(t)(syhjy)(0)
i=dydydτdτdτsy(t−τ−τ)hiy(τ)lix(τ)sy(−τ)hjy(τ).(A.7)
i

A.Appendix:OptimalMapformation
Exploitingthecorrelationassumptionsthisexpressionbecomes
µs2dydydτdτdτδ(y−y)δ(t−τ−τ+τ)hiy(τ)lix(τ)hjy(τ)
i=µs2dydτdτhiy(t−τ+τ)lix(τ)hjy(τ)
iyy=µs2dy((hilix)◦hj)(t)(A.8)
iwiththeopencircle◦denotingtheautocorrelationintegraldeﬁnedin(2.13).Wefocusonthe
secondtermintheright-handsideof(A.6).Thatis,
dydy(syηiylix)(t)(syηjy)(0)

i=dydydτdτdτsy(t−τ−τ)ηiy(τ)lix(τ)sy(−τ)ηjy(τ),(A.9)

iwhichsimpliﬁesto
µs2ση2maxdydydτdτdτδ(y−y)δ(t−τ−τ+τ)δijδ(y−y)δ(τ−τ)lix(τ)
<yy||i0<τ<tmax
=µs2ση2maxdydτdτδ(−τ+t)ljx(τ)
|y|<ymax
<t<τ0=µs2ση2maxdydτljx(t).(A.10)
0|<τy|<y<tmax
Altogethertheﬁnalexpressionsfortheexpectationvaluesbecome
∂sˆx(t)
sx(t)x=µs2hjx(−t)(A.11)
∂lj(t)
andsˆx(t)x=σχ2ljx(t)+ση2(µs2+σξ2)maxdydτljx(t)+(µs2+σξ2)dy(hiylix)◦hjy(t).
∂sˆx(t)
∂lj(t)0|y<τ|<y<tmaxi
(A.12)Equation(A.2)thereforetransformsinto
maxljx(t)σχ2+(µs2+σξ2)|y|<ymaxdydτση2(A.13)
<t<τ0+(µs2+σξ2)dy(hilix)◦hj(−t)(A.14)
yy
i=µs2hjx(−t).(A.15)
InsertingtheparameterσandτandapplyingaFouriertransformationﬁnallyleadsto(2.16).
97

A.Appendix:OptimalMapformation

Inordertotestwhethertheextremumisindeedaminimum,wehavetocalculatethesecond
readshwhicariation,v∂2[sx(t)−sˆx(t)]2
∂ljx(t)2
∂x∂sˆx(t)x∂sˆx(t)
=2∂ljx(t)s(t)∂ljx(t)−sˆ(t)∂ljx(t)
=0+2σχ2+ση2(µs2+σξ2)|y|<ymaxdydτ+
0<τ<tmax
(µs2+σξ2)dydτhjy(τ)2.(A.16)
Sincethesquaresarepositive,soisthesecondderivative,andthustheextremumisaminimum.

signalblurredGaussianA.4Inthissubsectionwepresentanequationequivalentto(2.16)ofthemaintextsoastoderivean
expressionforaGaussianblurredsignal.Asin(A.17),arealisticsignalwouldfulﬁllsomekindof
Gaussianrelationfortheexpectationvalue
sx(t)sx(t)
22
=Aexp−|x−x|exp−|t−t|.(A.17)
2σx22σt2
Forthiscasewecan,analogouslytoappendixA.3,deriveanequationlike(2.16).Sinceforthe
signaltheGaussiancorrelations,however,replacethedeltafunctions,e.g.,in(A.10),integralsover
spaceandtimecannotbeevaluateddirectly.Insteadtheycanonlyberestrictedtotheregion
wheretheGaussianisnon-negligible.Denotingthesetemporalandspatiallimitsbytandwe
canderivetheanalogueto(2.16),viz.,
22
ddtAexp−||exp−|t|2hjx+(t+t)
2σx22σt
=σχ2ljx(t)+ση2σξ2maxdydτljx(t)+σξ2dy(hiylix)◦hjy(t)
0|y<τ|<y<tmaxi
t2
+ση2|y|<ymaxdyddτAexp−2σ2ljx(t+t)
tmax<t<τ0+dyddtAexp−||2exp−t2((hiylix)◦hjy+)(t+t).(A.18)
22
i2σx2σt
Theeﬀectoftheadditionalremainingspatio-temporalintegralsascomparedto(2.16)isasmoothen-
ingoftheﬁnalreconstruction.Notonlyisthevalueataspeciﬁcpointinspaceandtime(y,t)
takenintoaccountbutneighboringpointsinanearlyareasurroundingitareincludedaswell.

A.Appendix:OptimalMapformation

PseudoinverseA.5ApseudoinversematrixBofanarbitrarym×nmatrixAwithitselements∈R,Cisageneralization
oftheinversematrix.FollowingMooreandPenrose[201,223]thepseudoinverseBisdeﬁnedby
equations,four

ABA=A,(A.19a)
BAB=B,(A.19b)
(AB)T=AB,(A.19c)
(BA)T=BA,(A.19d)
wheretheoverbardenotesthecomplexconjugation.ThepseudoinverseisusedforamatrixA
thatisofincompleterangandthereforecannotbeinverteddirectlybutonlyduetoauxiliary
constructions(A.19a,b).Thelasttwoequations(A.19c,d)tellusthattheproductofmatrixA
withitspseudoinverseBisHermitian.
Oneofthemostfamousapplicationsofthepseudoinverseistocalculatetheleast-squaresolution
ofasystemoflinearequations.

Ax=b→xˆ=Bb(A.20)
GivenamatrixAandavectorb,theabovesolutionxˆ=BbminimizestheEuclideannorm
||Axˆ−b||.Iftheinversematrixexists,thepseudoinversereducestothenormalinversematrix.
Forageneralreviewandapplicationsthereaderisreferredto,e.g.,[3].

B.Appendix:Lateral-linePerception

Incalculatethefollothewing,twweo-dimensionalbrieﬂyenlighﬂowteninathe(pretendedx,y)-planeparadox:whereasWhywecanwcannoteusecalculatecomplexanalysisanalyticallyto
withonlyrotational-symmetrictwocoordinatessettings,involve.g.ed.ﬂowaroundbodies,eventhoughthiscorrespondsalsotoasetup

B.1Planepotentialﬂow
Wedeﬁnethepotentialﬂowasasettingwheretheﬂowﬁeldvcanbecalculatedbymeansofthe
velocitypotentialΦ:Rn→Rthroughv=Φ.Inthesesettingsrotv=0;theﬂowﬁeldisvortex
free.Aspecialcaseofpotentialﬂowistheplanepotentialﬂow.Here,oneoftheﬂowﬁeldcom-
ponentsv=(vx,vy,vz)iszero,e.g.vz=0.Inaddition,theremainingtwocomponentsare
independentofthecoordinatewhichcorrespondstotheﬂowﬁeldcomponentthatiszero,e.g.,
v=(vx(x,y,t),vy(x,y,t),0).Incaseofplanepotentialﬂow,thefollowingstatementsholdtrue
forthepotentialΦandthestreamfunctionΨ;thelatterisdeﬁnedbymeansofvx=∂Ψ/∂yand
vy=−∂Ψ/∂x,
∂2Φ∂2Φ∂2Ψ∂2Ψ
∂x2+∂y2=0=∂x2+∂y2.(B.1)
Thatis,boththepotentialΦandthestreamfunctionΨsolvetheplaneLaplacianequationand
bothareconnectedthroughtheCauchy-Riemanndiﬀerentialequations,
∂Φ∂Ψ∂Φ∂Ψ
∂x=∂yand∂y=−∂x.(B.2)
Forthatreason,wecandeﬁneacomplexpotentialF(z):=Φ(x,y)+iΨ(x,y)withx+iy=z∈C
wherewecanderivethecomplexvelocityw(z),
dzdF=w(z)=vx(x,y)−ivy(x,y).(B.3)
Aswecanseeabove,itisalsocomposedofrealvaluedfunctions,namely,vxandvy.Indoingso,
wecananalyticallysolvemanyhydrodynamicproblemssuchastheﬂowaroundproﬁles.

B.2Rotational-symmetricpotentialﬂow
toThevariouscomplexhydrodescriptiondynamicofhproblems.ydroItdynamicisobsettinviousgsisthatquitonlyeptowwerfulaso-dimensionalitprovidesissuesancanbanalyticaleaddressedaccess
withcompaosedoftwone-dimensionalorealcocomplexordinatesfux,ynctionorr,φ.sinceInthetuitivelycomplex,onecocanordinatethinkofz=xextending+iy=rtheexp(iφcomplex)is
formalismtothree-dimensionsincaseofsymmetriesthateﬀectivelyreducethethree-dimensional
problemdemonstrateatbhandelowtoaexplicitlytwforo-dimensionalrotational-symmetricsetup.Howevﬂoer,w.itisnotpossibletodosoaswewill

101

B.Appendix:Lateral-linePerception

Thevelocityﬁeldforrotational-symmetricﬂowisgivenbyv=(vr(r,z,t),vz(r,z,t),0).The
continuityequationdivv=0nowreads
1∂(rvr)+∂cz=0.(B.4)
z∂r∂rThecontinuityequationholdstrueforastreamfunctiondeﬁnedbyrvr=−∂Ψ/∂zandrvz=
.r/∂Ψ∂Sincewearestudyingpotentialﬂowtheﬂowﬁeldhastobevortexfree,viz.,rotv=0.Thetwo
components(rotv)and(rotv)ofthecorrespondingvorticityrotvareobviouslyzero.According
tosymmetryreasons,zhowever,rthethirdcomponent(rotv)φisnotobviouslyzero.Thatis,wehave
toclaim2
(rotv)φ=∂∂vzr−∂∂vrz=−r1∂∂zΨ2+∂∂rr1∂∂rΨ=0.(B.5)
tIfwowecomequationsbineforthebconothtintheuitvyelocityequationpoten(B.4)tialΦandandrotvthe=str0eam(B.5)functionsimilarlyΨ,to(B.1)wecanderive
2222∂∂r2Φ+r1∂∂rΦ+∂∂z2Φ=0=∂∂r2Ψ−r1∂∂rΨ+∂∂z2Ψ.(B.6)
Aswecanseethelefthandsideandtherighthandsideof(B.6)arenotofthesametype.The
streamfunctiondoesnotfulﬁlltherotational-symmetricLaplacianequation,i.e,ΔΨ=0.
Inaddition,thestreamfunctionΨandthevelocitypotentialΦarenotconnectedthroughthe
Cauchy-Riemanndiﬀerentialequations(B.2)but
∂Φ1∂Ψ∂Φ1∂Ψ
∂r=−r∂zand∂z=r∂r.(B.7)
ThusimaginarywepartcannotofinterpretanalyticalthefunctionstreamF(z)function=Φ(Ψx,y)and+itheΨ(x,vyelo),citinypconcreto,otentialofΦtheasthecomplexrealvandelocitthye
potentialF(z).

Insummary,thestreamfunctionΨandthevelocitypotentialΦdonotsolvethesamediﬀerential
equation(B.6)–theLaplacianequation–and,inaddition,theyarenotconnectedbytheCauchy-
Riemanndiﬀerentialequations.Againstintuition,wethereforecannotusecomplexanalysisto
describerotational-symmetricﬂowandthusthree-dimensionalsetupsevenincaseofthisspecial
.symmetry

102

iMTendix:AppC.

C.1Optimalcombinationoftwomodalities
Theﬁringproﬁleofaneuronalmapcanbeinterpretedasthelikelihoodofﬁndinganobjectata
speciﬁcposition;seeSec.1.2intheintroduction.Inotherwords,neuronalmapsencodeposition
estimatorsbymeansoftheirﬁringproﬁles.Amodality,forinstance,auditionAorvisionV,can
berepresentedthroughGaussianﬁringproﬁles
fA/V(x):=SA/Vexp−(x−µA/V)2/(2σ2A/V)(C.1)
withmappositionxandestimatedposition,i.e.,meanµA/V,standarddeviationσA/V,and
maximalamplitudeSA/Voftheactivityproﬁles.Tocombinetwosuchestimatorstoamultimodal
estimateµMwehavetocalculateaweightedsumofthemeanvaluesµVandµAleadingto
µM=gVµV+gAµA(C.2)
withthenon-negativeweightsgVandgAfulﬁllinggV+gA=1.Thevarianceσ2Mofthecombined
readsestimatorσ2M=gV2σV2+(1−gV)2σA2.(C.3)
Next,wehavetochoosetheweightgVsoastominimizethevarianceσ2MwithrespecttogV;that
is,∂σM/∂gV=0.SolvingthelatterequationleadstogV=1−gA=σA2/(σA2+σV2)andﬁnallyto
σ2M=[σA2/(σA2+σV2)]2σV2+([σV2/(σA2+σV2)]2σA2=σV2σA2/(σA2+σV2).(C.4)
Wecaninserttheaboveresultinto(C.2)andcalculatetheoptimalcombinationscheme
µM=σA2/(σA2+σV2)µV+σV2/(σA2+σV2)µA.(C.5)
Formathematicaldetailsregardingthederivationseeelsewhere[276].
Theaboveformulaisabletodescriberesultsofneuronalprocessesbutwithoutexplaininghow
theyareachieved.Inalmostallcasesitispracticallyunknownhowtheneuronalsystemcomputes
anoptimalestimate.Inthecontextofneuronalmaps,however,asimplemultiplicationofthe
activityproﬁlesoftheinvolvedmapscanneuronallyrealizetheaboveoptimalcombinationscheme.
Themultimodalactivityproﬁleisthengivenby
fM(x)=SAexp−(x−µA)2/(2σA2)SVexp−(x−µV)2/(2σV2)

=SASVexp−1/2(µA−µV)2/σV2+σA2(C.6)
factorscaling2
exp−1/2σV2+σA2/σA2σV2x−σV2µA+σA2µV/σV2+σA2.(C.7)
1/σ2MµM

StandarddeviationσMandmeanµMofthemultimodalproﬁleareidenticaltotheoptimalvalues
wehavederivedabove;cf.(C.4)and(C.5).

103

iMTendix:AppC.

C.2Optimalcombinationofthreemodalities

WeextendtheoptimalcombinationschemefortwomodalitiesasderivedinSec.C.1tothree
modalities.ThemultimodalestimateµMofthreemodalities,say,auditionA,visionV,and
somatosensorySrepresentedbytheirmeanvaluesµA,µV,andµSwiththeirstandarddeviations
σA,σV,andσSisgivenby
σ2σ2σ2σ2σ2σ2
µM=VSµA+ASµV+AVµS,
σA2σV2+σA2σS2+σV2σS2σA2σV2+σA2σS2+σV2σS2σA2σV2+σA2σS2+σV2σS2
(C.8)

aconvexcombinationofµA,µV,andµS.ThecombinedmaximumlikelihoodestimateµMfeatures
thesmallestpossiblestandarddeviationσMgivenby
MσA2σV2+σA2σS2+σV2σS2
σ=σAσVσS.(C.9)
Thedistancebetweenthemultimodalestimateandaunimodalone,forinstanceaudition,isgiven
yb

104

AppC.iMTendix:

C.3ModelParameters
InTableC.1wesummarizethemodelparametersasusedinournumericalexperiments(seemain
text).

aluevparameternumberofmapneuronsN=100
synapseslearningminimalstrengthJmin=0.0
maximalstrengthJmax=0.25
maximalinitialstrength(J0)max=0.25
strengthofteachersynapsesJT=−1
timeonserespostsynapticptoinputspikeτTI=10ms
toteacherspikeτ=25ms
inputamplitudeS=50s−1
teacheramplitudeATA/V=/S100s−1
teacherwidthσT=0.025
shiftmaps–teacherΔ=0.2
learningtriallengthT=0.5s
simlearningulationtimparameterestepηΔt==3∙0.105−ms6
weightchangein
upupononinputoutputspikspikeewwout==1.−510.0
TableparametersC.1:ofMothedellearningparameterswindoaswusearedintakourennfromumerical[78]withexperimendiﬀeringtsor(seemainadditionaltext).parametersShapeandas
e.voabindicated

105

Bibliography

Bibliography[1]Abbott,L.F.&Nelson,S.B.(2000)Synapticplasticity:tamingthebeast,NatNeurosci3
1178–1183.[2]Alais,D.&Burr,D.(2004)TheVentriloquisteﬀectresultsfromnear-optimalbimodal
integration,CurrBiol14257–262.
[3]Albert,A.(1972)RegressionandtheMoore-Penrosepseudoinverse(Academic,NewYork).
[4]Amemiya,F.,Goris,R.C.,Masuda,Y.,Kishida,R.,etal.(1995)Thesurfacearchitectureof
snakeinfraredreceptororgans,BiomedRes-Tokyo16411–421.
[5]Amemiya,F.,Nakano,M.,Goris,R.C.,Kadota,T.,etal.(1999)Ultrastructureofthe
crotalinesnakeinfraredpitreceptors:Semconﬁrmationoftemﬁndings,AnatRec254
107–115.[6]Amemiya,F.,Tatsuo,U.,Goris,R.C.,&Kusunoki,Y.T.(1996)Ultrastructureofthecrotaline
snakeinfraredpitreceptors:Semconﬁrmationoftemﬁndings,AnatRec246135–146.
[7]Anastasio,T.J.,Patton,P.E.,&Belkacem-Boussaid,K.(2000)Usingbayesruletomodel
multisensoryenhancementinthesuperiorcolliculus,NeuralComp121165–1187.
[8]Bakken,G.S.(2007)AcritiqueofthermalmodelingofsnakeinfraredreceptionbyJones,
LynnandStone(2001),JTheorBiol244167–168.
[9]Bakken,G.S.&Krochmal,A.R.(2007)Theimagingpropertiesandsensitivityofthefacialpits
ofpitvipersasdeterminedbyopticalandheat-transferanalysis,JExpBiol2102801–2810.
[10]Bartels,M.,M¨unz,H.,&Claas,B.(1990)Representationofthelaterallineandelectrosensory
systemsinthemidbrainoftheaxolotl,Ambystomamexicanum,JCompPhysiolA167
347–356.[11]Beauchamp,M.S.,Argall,B.D.,Bodurka,J.,Duyn,J.H.,etal.(2004)Unravelingmultisensory
integration:patchyorganizationwithinhumanSTSmultisensorycortex,NatNeurosci7
1190–1192.[12]Ben-Israel,A.&Greville,T.N.E.(2003)GeneralizedInverses:TheoryandApplications
(Springer,NewYork,NY),2ndedition.
[13]Benson,D.L.,Colman,D.R.,&Huntley,G.W.(2001)Molecules,mapsandsynapsespeciﬁcity,
899–909.2NeurosciRevNat[14]Bergan,J.F.&Knudsen,E.I.(2009)Visualmodulationofauditoryresponsesintheowl
inferiorcolliculus,JNeurophysiol1012924–2933.
[15]Bi,G.Q.(2002)Spatiotemporalspecicityofsynapticplasticity:cellularrulesandmechanisms,
319–332.87ernCybBiol[16]Bi,DepG.Q.endence&Ponoo,spikeM.M.timing,(1998)synapticSynapticstrength,modiﬁcationsandpinostsynapticculturedcellhipptypoe,JcampalNeuroscineurons:18
472.464–1010[17]Bi,G.Q.&Poo,M.M.(2001)Synapticmodiﬁcationbycorrelatedactivity:Hebb’spostulate
revisited,AnnuRevNeurosci24139–166.
[18]Bialek,W.,Rieke,F.,deRuytervanSteveninck,R.R.,&Warland,D.(1991)Readinga
neuralcode,Science2521854–1857.
[19]Bisti,S.&Maﬀei,L.(1974)Behaviouralcontrastsensitivityofthecatinvariousvisual
meridians,JPhysiol241201–210.
[20]Blauert,J.(1999)SpatialHearing(MITPress,Cambridge,MA).
[21]Bleckmann,H.(1994)Receptionofhydrodynamicstimuliinaquaticandsemiaquaticanimals
her).(Fisc[22]Boll,S.F.(1979)Suppressionofacousticnoiseinspeechusingspectralsubtraction,IEEET
AcoustSpeech27113–20.
[23]Braucdynamicshi,S.,andOrio,Pelectroph.,&Latorre,ysiologicalR.analysis(2004)ofCluescoldtoreceptorunderstandingtrpm8,ProcoldcNatlsensation:AcadSciThermo-USA
499.494–1515101

107

Bibliography

[24]Braun,H.A.&Hensel,H.(1980)Staticanddynamicdischargepatternsofburstingcold
[25]ﬁbBraun,ersrelatedH.A.,Sctoh¨hafer,ypK.,othetical&Wissing,receptorH.mec(1984)hanisms,TheorienPﬂugundArchMoEurdelleJPhzumy¨ub3861–9.ertragungsver-
haltenthermosensitiverRezeptoren,FunktBiolMed326–36.
[26]Braun,H.B.,Dewald,M.,Sch¨afer,K.,Voigt,K.,etal.(1999)Low-dimensionaldynamicsin
sensorybiology2:Facialcoldreceptorsoftherat,JComputNeurosci717–32.
[27]Braun,H.B.,Huber,M.T.,Dewald,M.,Sch¨afer,K.,etal.(1998)Computersimulationsof
neuronalsignaltransduction:Theroleofnonlineardynamicsandnoise,IntJBifurcatChaos
881–889.8[28]Braun,W.,Eckhardt,B.,Braun,H.A.,&Huber,M.(2000)Phase-spacestructureofa
thermoreceptor,PhysRevE626352–6361.
[29]Bregman,A.S.(1990)AuditorySceneAnalysis(MITPress,Cambridge,MA).
[30]Bresciani,J.-P.,Dammeier,F.,&Ernst,M.O.(2008)Tri-modalintegrationofvisual,tactile
andauditorysignalsfortheperceptionofsequencesofevents,BrainResBul75(6)753–
760.[31]Bresciani,J.-P.,Ernst,M.O.,Drewing,K.,Bouyer,G.,etal.(2005)Feelingwhatyouhear:
auditorysignalscanmodulatetactiletapsperception,ExpBrainRes162172–180.
[32]vanBrunt,B.(2000)Thecalculusofvariations(Springer,Heidelberg).
[33]Bullock,T.H.&Diecke,F.P.J.(1956)Propertiesofaninfra-redreceptor,JPhysiol134
47–87.[34]Buonomano,D.V.&Merzenich,M.M.(2004)Corticalplasticity:Fromsynapsestomaps,
AnnuRevNeurosci21149–186.
[35]B¨urck,M.,Friedel,P.,Sichert,A.B.,Vossen,C.,etal.(2010)Optimalityinmono-and
multisensorymapformation,BiolCybern1031–20.
[36]B¨urck,M.&vanHemmen,J.L.(2007)Modelingthecochlearnucleus:Asiteformonaural
echosuppression?,JAcoustSocAm122(4)2226–2235.
[37]B¨urck,M.&vanHemmen,J.L.(2009)Neuronalidentiﬁcationofsignalperiodicitybybalanced
inhibition,BiolCybern100261–270.
[38]Campbell,A.L.,deCockBunning,T.,Stone,M.O.,Church,D.,etal.(1999)Surface
ultrastructureofpitorgan,spectacle,andnonpitorganepidermisofinfraredimagingboid
snakes:Ascanningprobeandscanningelectronmicroscopystudy,JStructBiol126105–120.
[39]vonCampenhausen,C.,Riess,I.,&Weissert,R.(1981)Detectionofstationaryobjectsby
theblindcaveﬁshAnoptichtysjordani,JCompPhysiolA143369–374.
[40]Carr,C.E.,Maler,L.,&Sas,E.(1982)Peripheralorganizationandcetralprojectionsofthe
electrosensorynervesingymnotiformﬁsh,JCompNeurol211139–153.
[41]Carr,R.W.,Pianova,S.,McKemy,D.D.,&Brock,J.A.(2009)Actionpotentialinitiation
intheperipheralterminalsofcold-sensitiveneuronsinnervatingtheguinea-pigcornea,J
1249–1264.587ysiolPh[42]Casseday,J.H.&Neﬀ,W.D.(1973)Psychophysicalstudiesofmonkeyvision:III.Spatial
luminancecontrastsensitivitytestsofmacaqueandhumanobservers,JAcoustSocAm
365–372.54(2)[43]Catania,K.C.,Leitch,D.B.,&Gauthier,D.(2009)Functionoftheappendagesintentacled
snakes(Erpetontentaculatus),JExpBiol213359–367.
[44]Claas,B.(1994)Removalofeyesinearlylarvalstagesalterstheresponseoftheclawedtoad,
[45]Clegg,XenopusJ.C.laevis,(1968)tosurfaceCalculuswaofves,PhvariationsysiolBeh(Oliv56erand423–428.Boyd,Edinburgh).
[46]Clifton,R.K.(1987)Breakdownofechosuppressionintheprecedenceeﬀect,JAcoustSoc
1834–1835.82(5)Am[47]deCockBuning,T.(1984)Atheoreticalapproachtotheheatsensitivepitorgansofsnakes,
509–529.111BiolTheorJ[48]deCockBuning,T.,Terashima,S.I.,&Goris,R.C.(1981)Crotalinepitorgansanalyzedas

108

Bibliography

warmreceptors,CellMolNeurobiol169–85.
[49]deCockBuning,T.,Terashima,S.I.,&Goris,R.C.(1981)Pythonpitorgansanalyzedas
warmreceptors,CellMolNeurobiol1271–278.
[50]Cohen,M.L.(1977)Measurementofthethermalpropertiesofhumanskin.Areview,JInvest
333–338.69Dermatol[51]Cohen,Y.E.&Knudsen,E.I.(1999)Mapsversusclusters:diﬀerentrepresentationsofauditory
spaceinthemidbrainandforebrain,TrendsNeurosci22128–135.
[52]Coombs,S.,Anderson,R.,Braun,C.B.,&Grosenbaugh,M.(2007)Thehydrodynamic
footprintofbenthic,sedentaryﬁshinundirectionalﬂow,JAcoustSocAm1221227–1237.
[53]StevCopenselli,Mand.,RoWebque,er-FecA.C.,hnerOlivlawseira,areR.F.,transfer&Kinoucfunctionshi,O.of(2002)excitablePhysicsmedia,ofPhpsycysRhophevEysics:65
901.60[54]´Cur˘ci´c-Blake,B.&vanNetten,S.M.(2006)Sourcelocationencodingintheﬁshlateralline
canal,JExpBiol2091548–1559.
[55]Dan,Y.&Poo,M.M.(2004)Spiketiming-dependentplasticityofneuralcircuits,Neuron44
23–30.[56]Davison,A.P.&Fr´egnac,Y.(2006)Learningcross-modalspatialtransformationsthrough
spiketiming-dependentplasticity,JNeurosci265604–5615.
[57]DeValois,R.,Morgan,H.,&Snodderly,D.(1974)Psychophysicalstudiesofmonkeyvision:
Iii.spatialluminancecontrastsensitivitytestsofmacaqueandhumanobservers,VisionRes
75–81.14[58]Dehaene,S.&Cohen,L.(2007)Culturalrecyclingofcorticalmaps,Neuron56384–398.
[59]Deneve,S.,Latham,P.E.,&Pouget,A.(1999)Readingpopulationcodes:aneuralimple-
mentationofidealobservers,NatNeurosci2(8)740–745.
[60]Deneve,S.,Latham,P.E.,&Pouget,A.(2001)Eﬃcientcomputationandcueintegration
withnoisypopulationcodes,NatNeurosci4(8)826–831.
[61]Deneve,S.&Pouget,A.(2004)Bayesianmultisensoryintegrationandcross-modalspatial
links,JPhysiology-Paris98249–258.
[62]Dr¨ager,U.C.&Hubel,D.H.(1975)Responsestovisualstimulationandrelationshipbetween
visual,auditory,andsomatosensoryinputsinmousesuperiorcolliculus,JNeurphysiol38
690–713.[63]Ebert,J.&Westhoﬀ,G.(2006)Behaviouralexaminationoftheinfraredsensitivityof
rattlesnakes(Crotalusatrox),JCompPhysiolA192941–947.
[64]Eckertf,E.R.G.&Carlson,WalterO.(1961)Naturalconvectioninanairlayerenclosed
betweentwoverticalplateswithdiﬀerenttemperatures,IntJHeatMassTransf2106–120.
[65]Eckhorn,R.,Bauer,R.,Jordan,W.,Brosch,M.,etal.(1988)Coherentoscillations:A
mechanismoffeaturelinkinginthevisualcortex?,BiolCybern60121–130.
[66]Ephraim,Y.&Malah,D.(2004)SpeechEnhancementUsingaMinimumMean-SquareError
Short-TimeSpectralAmplitudeEstimator,IEEETAcoustSpeech321109–21.
[67]Ernst,M.O.&Banks,M.S.(2002)Humansintegratevisualandhapticinformationina
statisticallyoptimalfashion,Nature415429–433.
[68]Ernst,M.O.,Banks,M.S.,&B¨ulthoﬀ,H.(2000)Touchcanchangevisualslantperception,
69–73.3NeurosciNat[69]Ernst,M.O.&B¨ulthoﬀ,H.(2004)Mergingthesensesintoarobustpercept,TrendsCognSci
162–169.8[70]Evans,L.C.(1998)PartialDiﬀerentialEquations,volume19ofGraduateSeriesinMathe-
matics(AmericanMathematicalSociety,Providence,RI).
[71]Faisal,A.A.,Selen,L.P.J.,&Wolpert,D.M.(2008)Noiseinthenervoussystem,NatRev
292–303.9Neurosci[72]Fiala,D.,Lomas,K.J.,&Stohrer,M.(1999)Acomputermodelofhumanthermoregulationfor
awiderangeofenvironmentalconditions:thepassivesystem,JApplPhysiol871957–1972.

109

Bibliography

[73]Flanagan,J.L.&Lummis,R.C.(1970)Signalprocessingtoreducemultipathdistortionin
smallrooms,JAcoustSocAm47(6)1475–81.
[74]Franosch,J.-M.P.,Lingenheil,M.,&vanHemmen,J.L.(2005)Howafrogcanlearnwhatis
whereinthedark,PhysRevLett95078106.
[75]Franosch,J.-M.P.,Sichert,A.B.,Suttner,M.D.,&vanHemmen,J.L.(2005)Estimating
positionandvelocityofasubmergedmovingobjectbytheclawedfrogXenopusandby
ﬁsh—Acyberneticapproach,BiolCybern93231–238.
[76]Franosch,J.-M.P.,Sobotka,M.C.,Elepfandt,A.,&vanHemmen,J.L.(2003)Minimalmodel
ofpreylocalizationthroughthelateral-linesystem,PhysRevLett91158101.
[77]Frens,M.A.&vanOpstal,A.J.(1998)Visual-auditoryinteractionsmodulatesaccade-related
activityinmonkeysuperiorcolliculus,BrainRes46211–224.
[78]Friedel,P.&vanHemmen,J.L.(2008)Inhibition,notexcitation,isthekeytomultimodal
sensoryintegration,BiolCybern98597–618.
[79]Furuya,K.&Kataoka,A.(2007)Robustspeechdereverberationusingmultichannelblind
deconvolutionwithspectralsubtraction,IEEETAudioSpeech151579–91.
[80]Galan,R.F.,Ermentrout,G.B.,&Urban,N.N.(2007)Reliabilityandstochasticsynchroniza-
tionintypeivs.typeiineuraloscillators,Neurocomputing702102–2106.
[81]Galan,R.F.,Fourcaud-Trocme,N.,Ermentrout,G.B.,&Urban,N.N.(2006)Correlation-
inducedsynchronizationofoscillationsinolfactorybulbneurons,JNeurosci263646–3655.
[82]Gancarz,G.&Grossberg,S.(1999)Aneuralmodelofsaccadiceyemovementcontrolexplains
task-speciﬁcadaptation,VisionRes393123–3143.
[83]Gardner,J.L.,Merriam,E.P.,Movshon,J.A.,&Heeger,D.J.(2008)Mapsofvisualspacein
humanoccipitalcortexareretinotopic,notspatiotopic,JNeurosci283988–3999.
[84]Geisler,C.D.(1990)FromSoundtoSynapse:PhysiologyoftheMammalianEar(Oxford
Oxford).Press,yersitUniv[85]Gelfand,I.M.&Fomin,S.V.(1963)Calculusofvariations(Prentice-Hall,EnglewoodCliﬀs,
NY).[86]Gelfand,J.J.,Pearson,J.C.,Spence,C.D.,&Sullivan,W.E.(1988)Multisensorintegration
inbiologicalsystems(IEEEPress,NewYork).
[87]Gerstner,W.,Kempter,R.,vanHemmen,J.L.,&Wagner,H.(1996)Aneuronallearning
ruleforsub-millisecondtemporalcoding,Nature38376–78.
[88]Gerstner,W.&Kistler,W.(2002)SpikingNeuronModels(CambridgeUniversityPress,
bridge).Cam[89]Gerstner,W.,Kreiter,A.K.,Markram,H.,&Herz,A.V.M.(1997)Neuralcodes:Firingrates
[90]andGillespie,beyond,B.W.,ProcMalvNatlar,AcadH.S.,Sci&USA94Florencio,12740–12D.A.F.741.(2001)Speechdereverberationvia
maximum-kurtosissubbandadaptiveﬁlltering,ProcIEEEIntConfAcoustSpeechSig-
3701–4.6cesspronal[91]Gonzalez,R.C.&Woods,R.E.(2002)DigitalImageProcessing(PrenticeHall,UpperSaddle
edition.2ndNJ),er,Riv[92]Goris,R.C.,Nakano,M.,Atobe,Y.,Kadota,T.,etal.(2000)Nervouscontrolofbloodﬂow
microkineticsintheinfraredorgansofpitvipers,AutonNeurosci-Basic8498–106.
[93]Goris,R.C.&Nomoto,M.(1967)InfraredReceptioninOrientalCrotalineSnake,Comp
BiochemPhysiol23879–892.
[94]Goris,ReceptorsR.C.in&aTCrotalineerashima,SnakS.I.e,T(1973)rimeresurusCentralFlaRespvovionseridis,toJExpInfra-redBiolStim58ulation59–76.ofthePit
[95]Goulet,J.,Engelmann,J.,Chagnaud,B.P.,Franosch,J.-M.P.,etal.(2008)Objectlocalization
throughthelaterallinesystemofﬁsh:Theoryandexperiment,JCompPhysiolA1941–17.
[96]Grace,M.S.,Woodward,O.M.,Church,D.R.,&Calisch,G.(2001)Preytargetingbythe
infrare-imagingsnakePython:eﬀectsofexperimentalandcongenitalvisualdeprivation,
BehavBrainRes11923–31.

110

Bibliography

[97]Gracheva,E.O.,Ingolia,N.T.,Kelly,Y.M.,Cordero-Morales,J.F.,etal.(2010)Molecular
basisofinfrareddetectionbysnakes,Nature4641006–1012.
[98]Gray,C.M.,K¨onig,P.,Engel,A.K.,&Singer,W.(1989)Oscillatoryresponsesincatvisual
cortexexhibitinter-columnarsynchronizationwhichreﬂectsglobalstimulusproperties,Nature
334–337.338[99]Grossbadaptiveerg,sS.,accadicRobeyerts,emoK.,vemenAguilar,tconM.,trol&byBullosupceriork,D.colliculus,(1997)AJneuralNeuroscimodel24ofm9706–9725.ultimodal
[100]Grothe,B.,Covey,E.,&Casseday,J.H.(2001)Medialsuperioroliveofthebigbrownbat:
Neuronalresponsestopuretones,amplitudemodulations,andpulsetrains,JNeurohysiol86
2219–2230.[101]Grothe,B.&Neuweiler,G.(2000)Thefunctionofthemedialsuperioroliveinsmallmammals:
[102]Gu,tempY.,oralAngelaki,receptiveD.E.,ﬁeldsin&auditoryDeAngelis,analysis,G.C.J(2008)CompNeuralPhysiolAcorrelates186of413–423.multisensorycue
integrationinmacaqueMSTd,NatNeurosci111201–1210.
[103]Gutfreund,Y.,Zheng,W.,&Knudsen,E.I.(2002)Gatedvisualinputtothecentralauditory
1556–1559.297Sciencesystem,[104]Hafed,Z.M.,Goﬀart,L.,&Krauzlis,R.J.(2008)Superiorcolliculusinactivationcausesstable
oﬀsetsineyepositionduringtracking,JNeurosci288124–8137.
[105]H¨ansel,E.&Schmidt,G.(editors)(2004)AcousticEchoandNoiseControl–APractical
Approach(Wiley,Hoboken).
[106]Harmening,W.M.,Nikolay,P.,Orlowski,J.,&Wagner,H.(2009)Spatialcontrastsensitivity
andgratingacuityofbarnowls,JVision9(7)1–12.
[107]Harris,G.G.(1964)Considerationsonthephysicsofsoundproductionbyﬁshes,inW.N.
Tavolga(editor),MarineBio-Acoustics,pp.233–247(PergamonPress,Oxford).
[108]Harris,G.G.,Flanagan,J.L.,&Watson,B.J.(1963)Binauralinteractionofaclickwitha
clickpair,JAcoustSocAm35672–678.
[109]Hartline,P.H.,Kass,L.,&Loop,M.S.(1978)MergingofmodalitiesintheopticTectum:
Infraredandvisualintegrationinrattlesnakes,Science1991225–1229.
[110]Hasan,K.M.,Salahuddin,S.,&Khan,R.M.(2004)Reducingsignal-biasfromMADestimated
noiselevelforDCTspeechenhancement,SignalProcess84151–62.
[111]Hassan,E.l.-S.(1985)Mathematicalanalysisofthestimulusforthelaterallineorgan,Biol
23–36.52ernCyb[112]Hebb,D.O.(1949)Theorganizationofbehavior:aneuropsychologicaltheory(Wiley,New
ork).Y[113]Helbig,H.B.&Ernst,M.O.(2007)Optimalintegrationofshapeinformationfromvisionand
touch,ExpBrainRes179595–606.
[114]Helmholtz,H.(1879)DieThatsacheninderWahrnehmung(AugustHirschwald,Berlin).
[115]vanHemmen,J.L.(2001)Theoryofsynapticplasticity,inFMoss&SGielen(editors),
HandbookofBiophysics,pp.771–823(Elsevier,Amsterdam),4thedition.
[116]vanHemmen,J.L.(2002)Themapinyourhead:Howdoesthebrainrepresenttheoutside
world?,ChemPhysChem3291–298.
[117]Hensel,H.(1975)StaticanddynamicactivityofwarmreceptorsinBoaconstrictor,Pﬂ¨ugers
191–199.353hivArc[118]Hensel,H.&Wurster,R.D.(1970)Staticpropertiesofcoldreceptorsinnasalareaofcats,J
271–275.33ysiolNeuroph[119]Herz,A.V.M.,Gollisch,T.,Machens,C.K.,&Jaeger,D.(2006)ModelingSingle-Neuron
dynamicsandComputations:ABalanceofDetailandAbstraction,Science31480–85.
[120]Hilis,J.M.,Ernst,M.O.,Banks,M.S.,&Landy,M.S.(2002)Combiningsensoryinformation:
Mandatoryfusionwithin,butnotbetween,senses,Science2981627–1630.
[121]Hisajima,T.,Kishida,R.,Atobe,Y.,Nakano,M.,etal.(2002)Distributionofmyelinated
andunmyelinatednerveﬁbersandtheirpossibleroleinbloodﬂowcontrolincrotalinesnake

111

Bibliography

infraredreceptororgans,JCompNeurol449319–329.
[122]H¨blindotting,hK.,umans:R¨Anosler,evF.,en&R¨t-relatedoder,pB.oten(2004)tialstudyAltered,ExpBrainauditory-tactileRes159interactions370–381.incongenitally
[123]Hubel,D.H.&Wiesel,T.N.(1977)Functionalarchitectureofmacaquemonkeyvisualcortex,
ProcRoySocLondB1981–59.
[124]Huber,M.T.&Braun,H.A.(2006)Stimulus-responsecurvesofaneuronalmodelfornoisy
subthresholdoscillationsandrelatedspikegeneration,PhysRevE730419291–04192910.
[125]opticsHughes,andA.retinal(1977)Thetoporganization.,ographyinofF.visionCrescitelliinmammals(editor),ofconHandbtrastingookoflifestsensoryyle:physiComparativologye,
volumeVII/5,chapter8,pp.613–756(Springer,BerlinHeidelbergNewYork).
[126]H¨urlimann,F.,Kiper,D.C.,&Carandini,M.(2002)Testingthebayesianmodelofperceived
speed,VisionRes422253–2257.
[127]Hyde,P.S.&Knudsen,E.I.(2001)Atopographicinstructivesignalguidestheadjustmentof
theauditoryspacemapintheoptictectum,JNeurosci218586–8593.
[128]Ito,M.(1989)Long-termdepression,AnnuRevNeurosci1285–102.
[129]Itti,L.&Koch,C.(2001)Computationalmodelingofvisualattention,NatRevNeurosci
194–203.2(3)[130]J¨arvilehto,M.(1985)Theeye:Visionandperception,inG.A.Kerkut&L.I.Gilbert(editors),
NervousSystem:Sensory,volume6ofComprehensiveInsectPhysiology,Biochemistry,and
Pharmacology,chapterFINDCHAPTER,pp.355–429(PergamonPress,Oxford).
[131]Jazayeri,M.&Movshon,J.A.(2006)Optimalrepresentationofsensoryinformationbyneural
populations,NatNeurosci9690–696.
[132]Jeﬀress,L.A.(1948)Aplacetheoryofsoundlocalization,JCompPhysiolPsychol4135–39.
[133]Jiang,B.,Trevi˜no,M.,&Kirkwood,A.(2007)Sequentialdevelopmentoflong-termpotentia-
tionanddepressionindiﬀerentlayersofthemousevisualcortex,JNeurosci279648–9652.
[134]Jiang,W.,Wallace,M.T.,Jiang,H.,Vaughan,J.W.,etal.(2001)TwoCorticalAreasMediate
MultisensoryIntegrationinSuperiorColliculusNeurons,JNeurophysiol85506–522.
[135]Johnson,D.H.&Dudgeon,D.E.(1993)ArraySignalProcessing:ConceptsandTechniques
(Prentice-Hall,UpperSaddleRiver,NJ).
[136]Johnson,K.O.,Hsiao,S.S.,&Yoshioka,T.(2002)Neuralcodingandthebasiclawof
psychophysics,Neuroscientist8111–121.
[137]Jones,B.S.,Lynn,W.F.,&Stone,M.O.(2001)Thermalmodelingofsnakeinfraredreception:
Evidenceforlimiteddetectionrange,JTheorBiol209201–211.
[138]Jost,J.&Li-Jost,X.(1998)Calculusofvariations(CambridgeUniversity,Cambridge).
[139]Kaas,J.H.&Collins,C.E.(2004)Theresurresctionofmultisensorycortexinprimates:
Connectionpatternsthatintegratemodalities,inG.Calvert,C.Spence,&B.E.Stein
(editors),TheHandbookofMultisensoryProcesses,chapter17,pp.285–293(MITPress,
MA).bridge,Cam[140]ofKadunce,Within-andD.C.,VCross-Moaughan,dalitJ.W.,yWSuppreallace,ssioninM.T.,theSupBenedek,eriorG.,etColliculus,al.J(1997)NeurophMecysiolhanisms78
2834–2847.[141]Kailath,T.(1974)Aviewofthreedecadesoflinearﬁlteringtheory,IEEETInformTheory
146–181.20[142]Kalmijn,A.J.(1988)Hydrodynamicandacousticﬁelddetection,inJ.Atema,R.R.Fay,A.N.
Popper,&W.N.Tavolga(editors),SensoryBiologyofAquaticAnimals,chapter4,pp.83–130
NY).ork,YNew(Springer,[143]Kandel,E.R.,Schwartz,J.H.,&Jessell,T.M.(2000)Principlesofneuralscience(McGraw-
Hill,NewYork),4th(international)edition.
[144]Kay,S.M.(1993)FundamentalsofStatisticalSignalProcessing(PrenticeHall,UpperSaddle
NJ).er,Riv[145]Kempter,R.,Gerstner,W.,&vanHemmen,J.L.(1999)Hebbianlearningandspikingneurons,

112

Bibliography

PhysRevE594498–4514.
[146]Kempter,R.,Leibold,C.,Wagner,H.,&vanHemmen,J.L.(2001)Formationoftemporal-
featuremapsbyaxonalpropagationofsynapticlearning,ProcNatlAcadSciUSA98
4166–4171.[147]King,A.J.(1999)Sensoryexperienceandtheformationofacomputationalmapofauditory
spaceinthebrain,Bioessays21900–911.
[148]King,A.J.(2009)Visualinﬂuencesonauditoryspatiallearning,PhilTransRSocB364
331–339.[149]King,A.J.&Hutchings,M.E.(1987)Spatialresponsepropertiesofacousticallyresponsive
neuronsinthesuperiorcolliculusoftheferret:Amapofauditoryspace,JNeurophysiol57
596–624.[150]inKing,theA.J.visual,Hutcandhings,auditoryM.E.,repreMosenore,tationsD.R.,in&theBlakemore,mammalianC.sup(1988)eriorDevcolliculus,elopmentalNatureplasticit332y
73–76.[151]King,A.J.&Palmer,A.R.(1983)Cellsresponsivetofree-ﬁeldauditorystimuliinguinea-pig
superiorcolliculus:Distributionandresponseproperties,JPhysiol342361–381.
[152]King,A.J.&Parsons,C.H.(1999)Improvedauditoryspatialacuityinvisuallydeprived
ferrets,EurJNeurosci11(11)3873–3883.
[153]Knierim,J.J.&vanEssen,D.C.(1992)Neuronalresponsestostatictexturepatternsinarea
V1ofthealertmacaquemonkey,JNeurophysiol67961–980.
[154]Knudsen,E.I.(1982)Auditoryandvisualmapsofspaceintheoptictectumoftheowl,J
1177–1194.2Neurosci[155]Knudsen,E.I.(1999)Mechanismsofexperience-dependentplasticityintheauditorylocaliza-
tionpathwayofthebarnowl,JCompPhysiolA185305–321.
[156]Knudsen,E.I.(2002)Instructedlearningintheauditorylocalizationpathwayofthebarn
owl,Nature417322–328.
[157]Knudsen,E.I.,Blasdel,G.G.,&Konishi,M.(1979)Soundlocalizationbythebarnowl(Tyto
alba)measuredwiththesearchcoiltechnique,JCompPhysiolA1331–11.
[158]Knudsen,E.I.&Brainard,M.S.(1991)Visualinstructionoftheneuralmapofauditoryspace
inthedevelopingoptictectum,Science25385–87.
[159]Knudsen,E.I.&Knudsen,P.F.(1985)Visionguidestheadjustmentofauditorylocalization
inyoungbarnowls,Science230545–548.
[160]Knudsen,E.I.&Knudsen,P.F.(1989)Visioncalibratessoundlocalizationindevelopingbarn
owls,JNeurosci93306–3313.
[161]Knudsen,E.I.,duLac,S.,&Esterly,S.D.(1987)Computationalmapsinthebrain,Annu
41–65.10NeurosciRev[162]Kobayashi,S.,Amemiya,F.,Kishida,R.,Goris,R.C.,etal.(1995)Somatosensoryandvisual
correlationintheoptictectumofapython,Pythonregius:ahorseradishperoxidaseand
golgistudy,NeurosciRes22315–323.
[163]Kobayashi,S.,Kishida,R.,Goris,R.C.,Yoshimoto,M.,etal.(1992)Visualandinfraredinput
tothesamedendriteinthetectumopticumofthepython,Pythonregius:electron-microscopic
350–352.597ResBrainevidence,[164]Koch,C.(1999)BiophysicsofComputation(OxfordUniversityPress,Oxford).
[165]K¨ording,K.&Wolpert,D.M.(2004)Bayesianintegrationinsensorimotorlearning,Nature
244–247.427[166]Krauzlis,R.J.,Basso,M.A.,&Wurtz,R.H.(1997)Sharedmotorerrorformultipleeye
movements,Science2761693–1695.
[167]Krauzlis,R.J.,Liston,D.,&Carello,C.D.(2004)Targetselectionandthesuperiorcolliculus:
Goals,choicesandhypotheses,VisionRes441445–1451.
[168]Krochmal,A.R.,Bakken,G.S.,&LaDuc,T.(2004)Heatinevolutionskitchen:evolutionary
perspectivesonthefunctionsandoriginofthefacialpitofpitvipers(viperidae:Crotalinae),

113

Bibliography

4231–4238.207BiolExpJ[169]Krueger,L.E.(1989)ReconcilingFechnerandStevens:Towardauniﬁedpsychophysicallaw,
BehavBrainSci12251–320.
[170]Kuttruﬀ,H.(1991)RoomAcoustics(ElsevierAppliedScience,LondonandNewYork),3rd
edition.[172][171]LamLaming,b,H.D.(1932)(1997)HydrTheoMedynamicsasurement(CamofbridgeSensationUniversit(OxfordyPress),Univ6thersityed.,Oxfedition.ord).
[173]Lebart,K.,Boucher,J.M.,&Denbigh,P.N.(2001)Anewmethodbasedonspectral
subtractionforspeechdereverberation,ActaAcoustica87(3)359–366.
[174]Leibold,C.,Kempter,R.,&vanHemmen,J.L.(2001)Temporalmapformationinthebarn
owlsbrain,PhysRevLett87248101.
[175]Li,Z.(2002)Asaliencymapinprimaryvisualcortex,TrendsCognSci6(1)9–16.
[176]Liman,E.R.(2006)ThermalgatingofTRPionchannels:Foodforthought?,SciStke326.
[177]Linden,D.J.&Connor,J.A.(1995)Long-termsynapticdepression,AnnuRevNeurosci18
319–357.[178]Lingenheil,M.(2004)TheoriederBeuteortungbeimKrallenfrosch,Master’sthesis,Technische
Universit¨atMu¨nchen.
[179]U.Luksch,Windhorst,H.(2008)&M.C.SensorimotorHirschin(editors),tegration:EncyclopOpticediaoftectum,NeurinM.osciencD.eBinder,(Springer,N.NewHirokYawork,a,
NY).[180]Luksch,H.,Gauger,B.,&Wagner,H.(2000)Acandidatepathwayforavisualinstructional
signaltothebarnowl’sauditorysystem,JNeurosci20RC70.
[181]Luo,L.&Flanagan,J.G.(2007)Developmentofcontinuousanddiscreteneuralmaps,Neuron
284–300.56[182]Lynn,G.W.(1931)Thestructureandfunctionofthefacialpitofthepitvipers,AmJAnat
97–133.49[183]Ma,W.J.,Beck,J.M.,Latham,P.E.,&Pouget,A.(2006)Bayesianinferencewithprobabilistic
populationcodes,NatNeurosci91432–1438.
[184]Mach,E.(1866)¨UberdiephysiologischeWirkungr¨aumlichvertheilterLichtreize,Sitzungsber
AkadWissWienII54393–408.
[185]Madison,D.V.,Malenka,R.C.,&Nicoll,R.A.(1991)Mechanismsunderlyinglong-term
potentiationofsynaptictransmission,AnnuRevNeurosci14379–397.
[186]Magosso,E.,Cuppinia,C.,Serino,A.,DiPellegrino,G.,etal.(2008)Atheoreticalstudy
ofmultisensoryintegrationinthesuperiorcolliculusbyaneuralnetworkmodel,Neural
Networks21817–829.
[187]Manley,G.A.,K¨oppl,C.,&Konishi,M.(1988)Aneuralmapofinterauralintensitydiﬀerences
inthebrainstemofthebarnowl,JNeurosci82665–2676.
[188]Markram,H.,L¨ubke,J.,Frotscher,M.,&Sakmann,B.(1997)Regulationofsynapticeﬃcacy
bycoincidenceofpostsynapticAPsandEPSPs,Science275213–215.
[189]McKemy,D.D.,Neuhausser,W.M.,&Julius,D.(2002)Identiﬁcationofacoldreceptor
revealsageneralroleforTRPchannelsinthermosensation,Nature41652–58.
[190]McLaughlin,T.&O’Leary,D.D.M(2005)Moleculargradientsanddevelopmentofretinotopic
maps,AnnuRevNeurosci28327–355.
[191]Meszler,R.M.,Auker,C.R.,&Carpenter,D.O(1981)Finestructureandorganizationofthe
infraredreceptorrelay,thelateraldescendingnucleusofthetrigeminalnerveinpitvipers,J
571–584.196NeurolComp[192]Middlebrooks,J.C.&Knudsen,E.I.(1984)Aneuralcodeforauditoryspaceinthecat’s
superiorcolliculus,JNeurosci42621–2634.
[193]Miller,K.(1970)Leastsquaresmethodsforill-posedproblemswithaprescribedbound,
SIAMJMathAnal152–74.
[194]Mills,A.W.(1958)Ontheminimumaudibleangle,JAcoustSocAm30237.

114

Bibliography

[195]Miyoshi,M.&Kaneda,Y.(1988)InverseFilteringofRoomAcoustics,IEEETAcoust
145–152.36(2)heecSp[196]Moiseenkova,V.,Bell,B.,Motamedi,M.,Wozniak,E.,etal.(2003)Wide-bandspectral
tuningofheatreceptorsinthepitorganofthecopperheadsnake(Crotalinae),AmJPhysiol
598–606.284[197]Molenaar,G.J.(1978)Thesensorytrigeminalsystemofasnakeinthepossessionofinfrared
receptors:II.Thecentralprojectionsofthetrigeminalnerve,JCompNeurol179137–152.
[198]&PMolenaar,.S.UlinskiG.J.(1992)(editors),AnatomySensorimotorandphInteysiologygrationof,infvraredolume17sensitivitofyBioloofgysnakofes,theinRC.eptiliaGans,
chapter5,pp.367–453(UniversityofChicagoPress,Chicago,IL).
[199]Molenaar,G.J.&Fizaan-Oostveen,J.L.F.P.(1980)Ascendingprojectionsfromthelateral
descendingandcommonsensorytrigeminalnucleiinpython,JCompNeurol189555–572.
[200]Mooney,R.D.,Klein,B.G.,&Rhoades,R.W.(1987)Eﬀectsofalteredvisualinputupon
thedevelopmentofthevisualsystemandsomatosensoryrepresentationsinthehamster’s
superiorcolliculus,Neurosci20537–555.
[201]Moore,E.H.(1920)Onthereciprocalofthegeneralalgebraicmatrix,BAmMathSoc26
394–395.[202]Morgan,M.L.,DeAngelis,G.C.,&Angelaki,D.E.(2008)Multisensoryintegrationinmacaque
visualcortexdependsoncuereliability,Neuron59662–673.
[203]Murphey,R.K.(1973)Mutualinhibitionandtheorganizationofanon-visualorientationin
Notonecta,JCompPhysiolA8431–40.
[204]Mysore,S.P.&Quartz,S.R.(2005)Modelingstructuralplasticityinthebarnowlauditory
localizationsystemwithaspike-timedependenthebbianlearningrule,ProcIEEEIntJoint
ConfNeuralNetworks52766–2771.
[205]Nakano,M.,Atobe,Y.,Goris,R.C.,Yazama,F.,etal.(2000)Ultrastructureofthecapillary
pericytesandtheexpressionofsmoothmuscleα-actinanddesmininthesnakeinfrared
sensoryorgans,AnatRec260299–307.
[206]Neely,StephenT.&Allen,JontB.(1979)Invertibilityofaroomimpulseresponse,JAcoust
165–9.66(1)AmcSo[207]Nelson,D.A.(1998)Invitededitorialon”penns1948paperrevisited,JApplPhysiol852–3.
[208]Nelson,M.E.&Bower,J.M.(1990)Brainmapsandparallelcomputers,TrendsNeurosci13
408.–403[209]Newman,E.A.,Gruberg,E.R.,&Hartline,P.H.(1980)Theinfraredtrigemino-tectalpathway
intherattlesnakeandinthepython,JCompNeurol191465–477.
[210]Newman,E.A.&Hartline,P.H.(1981)Integrationofvisualandinfraredinformationin
bimodalneuronsoftherattlesnakeoptictectum,Science213789–791.
[211]Newman,E.A.&Hartline,P.H.(1982)Theinfrared“vision”ofsnakes,SciAm246(3)
98–107.[212]Nilius,B.,Talavera,K.,Owsianik,G.,Prenen,J.,etal.(2005)GatingofTRPchannels:a
voltageconnection?,JPhysiol1235–44.
[213]Norwich,K.H.&Wong,W.(1997)Uniﬁcationofpsychologicalphenomena:Thecomplete
formofFechner’slaw,PercPsychophys59929–940.
[214]Olsen,J.F.,Knudsen,E.I.,&Esterly,S.D.(1989)Neuralmapsofinterauraltimeandintensity
diﬀerencesintheoptictectumofthebarnowl,JNeurosci92591–2605.
[215]Oppenheim,A.V.,Schafer,R.W.,&Stockham,T.G.(1968)Nonlinearﬁlteringofmultiplied
andconvolvedsignals,ProcIEEE56(8)1264–91.
[216]vanOpstal,A.J.&Munoz,D.P.(2004)Auditory-visualinteractionssubservingprimategaze
orienting,inG.Calvert,C.Spence,&B.E.Stein(editors),TheHandbookofMultisensory
Processes,chapter23,pp.373–393(MIT,Cambridge,MA).
[217]Oˇguzt¨oreli,M.N.&Caelli,T.M.(1985)Aninverseprobleminneuralprocessing,BiolCybern
239–245.53

115

Bibliography

[218]Panton,R.L.(2005)Incompressibleﬂow(Wiley),3rdedition.
[219]Pneuronsappas,fromT.C.,pitvipMotamedi,erM.,thermosensors,&Christensen,AmJPhB.N.ysiolCell(2004)PhysiolUnique287temp1219–1228.erature-activated
[220]Patton,P.E.,Belkacem-Boussaid,K.,&Anastasio,T.J.(2002)Multimodalityinthesuperior
colliculus:aninformationtheoreticanalysis,CognitiveBrainRes1410–19.
[221]aPecka,neuronalM.,netZahn,workT.Pfor.,soundSaunier-Reblocalizationori,B.,inSivreveke,erbI.,eranettal.environ(2007)ments.,InhibitingJtheNeurosci.inhibition:27(7)
1782–1790.[222]Pennes,H.H.(1948)Analysisoftissueandarterialbloodtemperaturesintherestinghuman
forearm,JApplPhysiol193–122.
[223]Penrose,R.(1955)Ageneralizedinverseformatrices,CambridgePhilosSoc51406–413.
[224]PNeuronserrault,UseT.J.,VDistinctaughan,OpJ.W.,erationalStein,ModesB.E.,in&theWInallace,tegrationM.T.of(2005)MultisensorySuperiorStimColliculusuli,J
2575–2586.93ysiolNeuroph[225]Pickles,J.O.(1988)AnIntroductiontothePhysiologyofhearing(AcademicPress,London),
edition.2nd[226]Planck,M.(1901)Ueberdasgesetzderenergieverteilungimnormalspectrum,AnnPhys309
553–563.[227]Pouget,A.,Dayan,P.,&Zemel,R.S.(2003)Inferenceandcomputationwithpopulation
codes,AnnuRevNeurosci26381–410.
[228]Press,W.H.,Teukolsky,S.A.,Vetterling,W.T.,&Flannery,B.P.(2007)NumericalRecipes:
TheArtofScientiﬁcComputing,3rded.(CambridgeUniversity,Cambridge).
[229]Puetter,R.C.,Gosnell,T.R..,&AmosYahil,A.(2005)Digitalimagereconstruction:Deblur-
ringanddenoising,AnnuRevAstronAstrophys43139–194.
[230]Putzar,L.,Goerendt,I.,Lange,K.,R¨osler,F.,etal.(2007)Earlyvisualdeprivationimpairs
multisensoryinteractionsinhumans,NatNeurosci101243–1245.
[231]Rauschecker,J.P.&Kniepert,U.(1994)Auditorylocalizationbehaviourinvisuallydeprived
cats,EurJNeurosci6(1)149–160.
[232]deRivaz,Peter&Kingsbury,Nick(2001)Bayesianimagedeconvolutionanddenoisingusing
complexwavelets,IEEEImageProc2273–276.
[233]R¨oder,B.,R¨osler,F.,&Spence,C.(2004)Earlyvisionimpairstactileperceptionintheblind,
121–124.14BiolCurr[234]Roder,B.,Teder-Salejarvi,W.,Sterr,A.,Rosler,F.,etal.(1999)Improvedauditoryspatial
tuninginblindhumans,Nature400162–166.
[235]Roper,P.,Bressloﬀ,P.C.,&Longtin,A.(2000)Aphasemodeloftemperature-dependent
mammaliancoldreceptor,NeuralComput121067–1093.
[236]Rosenfeld,D.(2002)NewApproachtoGriddingUsingRegularization,MagnResonMed48
193–202.[237]Rowland,B.A.,Quessy,S.,Stanford,T.R.,&Stein,B.E.(2007)Multisensoryintegration
shortensphysiologicalresponselatencies,JNeurosci275879–5884.
[238]Rowland,B.A.,Stanford,T.R.,&Stein,B.E.(2007)Abayesianmodeluniﬁesmultisensory
spatiallocalizationwiththephysiologicalpropertiesofthesuperiorcolliculus,ExpBrainRes
153–161.180[239]Rowland,B.A.,Stanford,T.R.,&Stein,B.E.(2007)Amodeloftheneuralmechanisms
underlyingmultisensoryintegrationinthesuperiorcolliculus,Perception36(10)1431–1443.
[240]Rucci,M.,Edelman,G.M.,&Wray,J.(1999)Adaptionoforientingbehavior:Fromthebarn
owltoaroboticsystem,IEEETRobot1596–110.
[241]Rucci,M.,Tononi,G.,&Edelman,G.M.(1997)Registrationofneuralmapsthroughvalue-
dependentlearning:Modelingthealignmentofauditoryandvisualmapsinthebarnowl‘s
optictectum,JNeurosci17334–352.
[242]Sarkar,T.K.,Weiner,D.D.,&Jain,V.K.(1981)SomemathematicalConsiderationsin

116

Bibliography

DealingwiththeInverseProblem,IEEETAntennPropag29373–379.
[243]Sch¨afer,K.,Braun,H.A.,&Hensel,H.(1982)Staticanddynamicactivityofcoldreceptors
atvariouscalciumlevels,JNeurophysiol471017–1028.
[244]Sch¨afer,K.,Braun,H.A.,&K¨urten,L.(1988)Analysisofcoldandwarmreceptoractivityin
vampirebatsandmice,PﬂugArchEurJPhy412188–194.
[245]Schreiner,C.E.&Langner,G.(1988)Periodicitycodingintheinferiorcolliculusofthecat.
II.Topograhicalorganization,JNeurophysiol601823–1840.
[246]Schroeder,D.M.&Loop,M.S.(1976)Trigeminalprojectionsinsnakespossessinginfrared
sensitivity,JCompNeurol1691–14.
[247]Sejnowski,T.J.(1995)Timeforanewneuralcode?,Nature37621–22.
[248]Seung,H.S.&Sompolinsky,H.(1993)Simplemodelsforreadingneuronalpopulationcodes,
ProcNatlAcadSciUSA9010749–10753.
[249]Shams,L.,Kamitani,Y.,&Shimojo,S.(2000)Whatyouseeiswhatyouhear,Nature408
788.[250]Shumway,C.A.(1989)Multipleelectrosensorymapsinthemedullaofweaklyelectricgymno-
tiformﬁsh.I.Physiologicaldiﬀerences,JNeurosci94388–4399.
[251]Sichert,A.B.,Bamler,R.,&vanHemmen,J.L.(2009)Hydrodynamicobjectrecognition:
Whenmultipolescount,PhysRevLett102058104.
[252]Sichert,A.B.,Friedel,P.,&vanHemmen,J.L.(2006)Snake’sperspectiveonheat:Recon-
structionofinputusinganimperfectdetectionsystem,PhysRevLett9768105.
[253]Sichert,A.B.&vanHemmen,J.L.(2010)Howstimulusshapeaﬀectslateral-lineperception:
analyticalapproachtoanalyzenaturalstimulicharacteristics,BiolCybern102177–180.
[254]Song,S.,Miller,K.D.,&Abbott,L.F.(2000)Competitivehebbianlearningthroughspike-
timing-dependentplasticity,NatNeurosci3919–926.
[255]Stanford,L.R.&Hartline,P.H.(1980)Spatialsharpeningbysecond-ordertrigeminalneurons
incrotalineinfraredsystem,BrainRes185115–123.
[256]Stanford,L.R.,Schroeder,D.M.,&Hartline,P.H.(1981)Theascendingprojectionofthe
nucleusofthelateraldescendingtrigeminaltract:Anucleusintheinfraredsystemofthe
rattlesnake,Crotalusviridis,JCompNeurol201161–173.
[257]Stavenga,D.G.(2002)Reﬂectionsoncolourfulommatidiaofbutterﬂyeyes,JExpBiol205
1077–1085.[258]Steenken,R.,Colonius,H.,Diederich,A.,&Rach,S.(2008)Visualandauditoryinteraction
insaccadicreactiontime:Eﬀectsofauditorymaskerlevel,BrainRes1220150–156.
[259]Stein,B.E.&Gaither,N.S.(1981)Sensoryrepresentationsinreptilianoptictectum:Some
comparisonswithmammals,JCompNeurol20269–87.
[260]Stein,B.E.,Jiang,W.,&Stanford,T.R.(2004)Multisensoryintegrationinsingleneuronsof
themidbrain,inG.Calvert,C.Spence,&B.E.Stein(editors),TheHandbookofMultisensory
Processes,chapter15,pp.243–264(MIT,Cambridge,MA).
[261]Stein,B.E.&Meredith,M.A.(1993)TheMergingoftheSenses(MITPress,Cambridge,
MA).[262]Stein,B.E.&Stanford,T.R.(2008)Multisensoryintegration:Currentissuesfromthe
perspectiveofthesingleneuron,NatRevNeurosci9255–266.
[263]Stryer,L.(1986)Cyclicgmpcascadeofvision,AnnualReviewofNeuroscience987–119.
[264]Sullivan,W.E.&Konishi,M.(1986)Neuralmapofinterauralphasediﬀerenceintheowl’s
brainstem,ProcNatlAcadSciUSA838400–8404.
[265]Takahashi,T.&Konishi,M.(1986)Selectivityforinterauraltimediﬀerenceintheowl’s
midbrain,JNeurosci63413–3422.
[266]Terashima,S.I.,Goris,R.C.,&Katsuki,Y.(1970)Structureofwarmﬁberterminalsinthe
pitmembraneofvipers,JUltraMolStructR31494–506.
[267]Thomson,R.F.(2001)DasGehirn(Spektrum,Akad.Verl.,Heidelberg,Berlin),3rdedition.
[268]Tikhonov,A.N.,Arsenin,V.Y.,&John,F.(1977)SolutionofIll-PosedProblems(V.H.

117

Bibliography

Winston&Sons,Washington,DC),englishtranslationoftheoriginalRussiantext.
[269]forTikhonotheSv,olA.N.,utionofGoncIll-Poseharskyd,PrA.V.,oblemsStepano(Kluv,werV.V.,&AcademicYagola,A.G.Publishers,(1995)DordrecNumerichalt),Methoenglishds
translationoftheoriginalRussiantext.
[270]Truckenbrodt,E.(1999)Fluidmechanik2(Springer),4thedition.
[271]Udin,S.&Fawcett,J.W.(1988)Formationoftopographicmaps,AnnuRevNeurosci11
289–327.[272]Unoki,M.,Sakata,K.,Furukawa,M.,&Akagi,M.(2004)Aspeechdereverberationmethod
basedontheMTFconceptinpowerenveloperestoration,JAcousticalScienceandTechnology
243–254.25(4)[273]Unoki,Masashi,Furukawa,Masakazu,&Sakata,KeigoanfAkagi,Masato(2004)Animproved
methodbasedontheMTFconceptforrestoringthepowerenvelopefromareverberantsignal,
JAcousticalScienceandTechnology25(4)232–42.
[274]theUrsino,supM.,eriorCuppinia,colliculus:C.,aneuralMagosso,netwE.,orkmoSerino,del,JA.,etComputal.(2009)NeurosciMultisensory26(1)55–73.integrationin
[275]Voets,T.,Droogmas,G.,Wissenbach,U.,Janssens,A.,etal.(2004)Theprincipleof
temperature-dependentgatingincold-andheat-sensitiveTRPchannels,Nature430748–
754.[276]vanderWaerden,B.L.(1957)MathematischeStatistik(Springer,Berlin).
[277]Wallace,M.T.,Carriere,B.N.,PerraultJr,T.J.,Vaughan,J.W.,etal.(2006)Thedevelopment
ofcorticalmultisensoryintegration,JNeurosci26(46)11844–11849.
[278]Wallace,M.T.,PerraultJr,T.J.,Hairston,W.D.,&Stein,B.E.(2004)Visualexperienceis
necessaryforthedevelopmentofmultisensoryintegration,JNeurosci249580–9584.
[279]Wallace,M.T.&Stein,B.E.(1997)Developmentofmultisensoryneuronsandmultisensory
integrationincatsuperiorcolliculus,JNeurosci172429–2444.
[280]Wallace,M.T.&Stein,B.E.(2007)Earlyexperiencedetermineshowthesenseswillinteract,
921–926.97ysiolNeurophJ[281]Wandell,B.A.(1995)FoundationsofVision(SinauerAssociates,Sunderland,MA).
[282]Wehr,Michael&Zador,AnthonyM.(2003)Balancedinhibitionunderliestuningandsharpens
spiketiminginauditorycortex,Nature426442–446.
[283]Wickesberg,R.E.&Oertel,D.(1990)Delayed,frequency-speciﬁcinhibitioninthecochlear
nucleiofmice:Amechanismformonauralechosuppression,JNeurosci101762–1768.
[284]Willshaw,D.J.&Malsburg,C.(1976)Howpatternedneuralconnectionscanbesetupby
self-organization,ProcRoySocLondB194431–445.
[285]Winkowski,D.E.&Knudsen,E.I.(2006)Top-downgaincontroloftheauditoryspacemap
bygazecontrolcircuitryinthebarnowl,Nature439336–339.
[286]Wissler,E.H.(1998)’penns1948paperrevisited,JApplPhysiol8535–41.
[287]Witten,I.B.,Knudsen,E.I.,&Sompolinsky,H.(2008)AHebbianlearningrulemediates
asymmetricplasticityinaligningsensoryrepresentations,JNeurophysiol1001067–1079.
[288]Wu,Mingyang&Wang,DeLiang(2006)Atwo-stagealgorithmforone-microphonereverberant
speechenhancement,IEEETAudioSpeech14774–84.
[289]Yang,Y.,Chen,J.,Engel,J.,Pandya,S.,etal.(2006)Distanttouchhydrodynamicimaging
withanartiﬁciallateralline,ProcNatlAcadSciUSA10318891–18895.
[290]Yegnanarayana,B.,Avendano,C.,Hermansky,H.,&SatyanarayanaMurthy,P.(1999)
Speechenhancementusinglinearpredictionresidual,SpeechCommun2825–42.
[291]Yegnanarayana,B.&SatyanarayanaMurthy,P.(2000)EnhancementofReverberantSpeech
UsingLPResidual,IEEETSpeechAudioP8267–81.
[292]Yost,W.A.(1991)Auditoryimageperceptionandanalysis:Thebasisforhearing,HearRes
8–18.56(7)[293]Zeil,J.&Hemmi,J.M.(2006)Thevisualecologyofﬁddlercrabs,JCompPhysiolA192
1–25.