Read anywhere, anytime
Description
Subjects
Informations
Published by | technische_universitat_munchen |
Published | 01 January 2010 |
Reads | 13 |
Language | English |
Document size | 23 MB |
Exrait
Predicting
Maps,
rySenso
yitOptimal,
eatureF
Andreas
rmanceerfoP
Extraction,
B.
Sichert
nda
rningLea
Photograph
c
courtesy
Coverillustration:Photographccourtesy
esthoffWGuidoofTitle:Autor:PredictingAndreasB.SicSensoryhertPerformance
Subtitle:Maps,Optimality,FeatureExtraction,
Learning
and
FundinghasbeenprovidedbytheBernsteinCenter
theforCoTeSysComputationalExzellenzNeuroscCluster,ienceand(BCCN)the–Sloan-SwMunich,artz
ion.taoundF
PredictingSensoryPerformance
Maps,Optimality,FeatureExtraction,andLearning
B.AndreasSichert
Vollst¨andigerAbdruckdervonderFakult¨atf¨urPhysikderTechnischenUniversit¨atM¨unchen
zurErlangungdesakademischenGradeseinesDoktorsderNaturwissenschaftengenehmigten
Dissertation.
Vorsitzender:Univ.-Prof.Dr.Th.Hugel
Pr¨uferderDissertation:1.Univ.-Prof.Dr.J.L.vanHemmen
NetzR.Dr.Univ.-Prof.2.
DieDissertationwurdeam14.07.2010beiderTechnischenUniversit¨atM¨uncheneingereichtund
durchdieFakult¨atf¨urPhysikam20.09.2010angenommen.
Preface
Wow,complexyouareprorecess.adingItmis,yhoPhDwevtheser,ais!goIodmean,eyxampleouareofwhatreading–makeswhichneurisosanciencealmostsounbelievfascinating.ably
Let’stakeacloserlookattheaboveexampleofreadingtounderstandwhatneuroscienceisall
out.abAtfirstglance,readingstartswithsometinyblackdotsonawhitesheetofpaperwhichform
geometricalstructuresthatwedenoteasletters.Wecanperceivethembecauseoureyesandthe
subsequentneuronalprocessingofthevisualsystemareabletotranslatepicturesintheexternal
worldintoneuronalinformationthatformsacorrespondingrepresentationwithinourbrain.The
visualsystem,however,doesnotsettleinformationprocessingbypresentingcloudsofblackdots.
Atgroupthisthepoint,ineuronalnformationrepreseprontationcessingofstartsthesetocloudsgetreallyofdotstosophisticated.mentalobjeCognitivcts–enetheuronalletters.circuitsThe
activitiesindividualetc.lettersToarefinallyrelategaindatoeacreasonablehothermetoaningform,wtheordsbrainthatusestherepresenundterreallyingobjectsgrammar,attributes,thatis
thespecificmethodologyoftheparticularlanguagetobuildsentencesoutofindividualwords.
andForsenanytencesonearewhoisonlynotalreadymeaningfulbimpressedecauseletwemeprelateointtheoutsetwocloudsfurtherofdotsissues.andFirst,thecorrespletters,wondingords,
abstractmentalobjectstopastsensoryexperience.Wecaninterprettheword“barking”because
oursomevisualtimesysago.temThisrelatesisduethetoblacthekfactdotsonthatthediffsheeterentofmopapdalitieseratarehandabtoletotheinsoundteractofafaithfudoglly,heardfor
instance,bymeansofcombiningthecorrectmemoriesofvisualandauditorysensoryevents.
Second,theanswertothequestionofwheresuchintelligentneuronalcircuitscomefromis
elusive.Mayevolutionbeananswer,viz.,aretheseneuronalstructuresencodedinourgenes?
No,notinthefirstplace.OtherwisetherewouldbegenesforBavarian,German,English,and
otherlanguages.Theabilitytoread,tointerpretabstractmethodologyandsymbols,islearned.
Moreover,thereisnoteacherthatgivesbabieslecturesintheirmothertongue.So,ourbrain
teachesitself,thatisitlearnspurelyfromsensoryexperiencetobuilditselfinawaytobebest
suitedforComparingprocessingthesesensoryabilities–expourerience.abilities–withthoseofman-madeandsocalled“cognitive”
Fromsystemsmyispersonalimpressivpe.erspInectivconethtrastatistowhnatureatmakweeshatheve,fieldhowofever,justneurosciencestartedsotoinfascinating:ventsuchDiscovsystems.ering
thenervoussystem,ahellofamachinery!
ofdiffTheerentotolsfordisciplinesdiscovisneringethecessarynervtoousenlighsystemtenthisarecomplexmanifold.topic.ItturnsByoutmeansthatofathisbroadthesisintewreaclotionok
atsituatedthenervatoustheinsystemtersectionthroughofthebiology,glassesofmedicine,theoreticandalphysicsneuroscience.employingTheoreticalmathematicalneurosciencetools.Inis
mashort,jorthisdrivingdissertationforceforfothecuseswholeon“sPystemredicting–includingSensoryPitsdeverformance”elopment.sinceForsensorythatpuexprperieose,nceweiswillthe
themderivetospecifimathematicalcsettings,viz.,descriptionsdiffoferentsensorysenses.systems,featureextractionandlearning,andapply
v
Preface
1ChapterThefirstchaptergivesanintroductiontofundamentalconceptsofsensoryprocessingsuchasthat
ofaneuronalmapwhichareneededtounderstandtherestofthework.Besides,thereferences
giveninthecompactintroductionshouldprovidethereaderwithagoodstartingpointforown
studies.
2ChapterInordertoexploittheinformationprovidedbysensorysystems,animalsreconstructthespatio-
temporalenvironmentfromtheirsensorysystems.Inthischapterwederiveauniversalframework
thatallowstocalculatethespecificlayoutoftheinvolvedneuronalnetworkbymeansofa
generaltransmissionmathematicalandrudimenprinciple,talknoviz.,wledgestocofhasticthedetectionoptimality.procThatess,is,thisgivenapproacaknohwnallophwstoysicalestimatesignal
thepossibleperformanceandtopredictneuronalpropertiesofbiologicalsensorysystems.
3ChapterInthischapterweapplyourframeworkderivedinChap.2totheauditorysystem.Thatis,
comingfromthegeneralmathematicalprincipleofoptimalitywededuceanarchetypemodeland
correspondingtemporalreceptivefieldsforauditorysignalenhancementandechosuppression.We
comparetheperformanceofboththeneuronalandthemathematicalimplementationofthemodel
tostate-of-the-artalgorithmsandbiologicalsolutions.
4ChapterThelateral-linesystemisauniquemechano-sensoryfacilityofaquaticanimalsthatenablesthem
notonlytolocalizeprey,predators,obstacles,andconspecificsbutalsotorecognizehydrodynamic
objectsaswedemonstrateinthischapter.Wepresentbothananalyticapproachthatshowsthe
effectofsubmergedmovingobjectshapeandanexplicitmodelexplaininghowaquaticanimalscan
distinguishdifferentshapes.Furthermore,wepredictnaturallimitationsofthissensorycapability.
5ChapterTwogroupsofsnakespossessaninfrareddetectionsystemtoperceiveaheat-imageoftheir
environment.Thischapterexploresthedifferentstagesoftheinfraredpathwaysuchastransmission,
detection,andneuronalen-anddecodinginordertoprovidefundamentalinsightstothissensory
system.Thethoroughphysicaldescriptionoftheinvolvedphenomenaaswellastheincorporation
ofanatomicaldataintoanexplicitmodelextendthepresentpictureofthissensorysystem.
6ChapterForneuronalprocessingofsensoryinformationuni-andmultimodalmapsplayakeyrole.The
preciseinteractionofsuchmapsisnotpresentatbirthbuthastobelearnedduringthedevelopment
ofananimalbymeansofsensoryexperience.Thischapterpresentstheconceptofintegrated
MultimodalTeachingthatcombinessupervisedspike-timing-dependentplasticityandmultimodal
integrationtoinherentlydevelopafaithfulinteractionofdifferentsensorymodalities.
vi
yThankou
Preface
ManypeoplehavesupportedmeduringmytimeatthePhysikDepartmentoftheTechnische
Universit¨atM¨unchen.Inthefirstplace,IthankLeovanHemmen,mythesissupervisor.Hehas
introducedmetothetopicoftheoreticalneuroscience,viz.,biophysics.Hisdedicationtoscientific
researchisveryimpressive,andthroughmanyinspiringdiscussionsIhavelearnedalotabout
science.
Writingthisthesiswouldnothavebeenpossiblewithoutthepleasantinteractionandcooperation
mawithysamyythankcolleaguesyoutoattheMoritzB“T35”¨urcck,hairStefanieandaB¨urcfewk,pReopleolandbeyUtz,ond.InAndreasthecVonollmatextyr,ofandthisthesis,ChristineI
Voßenwhohaveproofreadpartsofthethesisandhavegivenhelpfulcomments.
Asawhole,Iwasgiventheopportunitytomakemanyvaluableexperiencesduringthetimeof
mUnivyPhDersity.wIork;havinealsoaddition,meteIwxceptionasablealptoeoplejoinandthehaBavyeeriscmadehenewEliteAkfriends.ademieForandalloftothisvisitIamHarvvardery
grateful.
Finally,Ithankmyfriends,myparents,mybrother,andBabsi,whohavealwaysbeenthereto
me.ortsupp
2010Juneh,Munic
vii
Contents
PrefacetstenConInformationSensorycessingPro11.1Stepsofsensoryprocessing...............................
1.1.1Signalmapping..................................
1.1.2Sensoryencoding.................................
1.1.3Neuronaldecoding................................
1.2Neuronalmaps......................................
1.2.1Mapproperties..................................
1.2.2Collectionofcomputationalmaps........................
1.2.3Mapdevelopment................................
1.3Learninginsensorysystems...............................
1.3.1Localmechanisms................................
1.3.2Globalstructures.................................
ormationFMapOptimal22.1Startingpoint.......................................
2.2Mathematicalmodel...................................
2.2.1Definitionoftheproblem............................
2.2.2Optimalreconstruction.............................
2.2.3Matrixnotation.................................
2.3Relationtocommonmethods..............................
2.3.1Pseudo-inverse...................................
2.3.2Maximum-likelihoodapproach..........................
2.4Neuronalrealizationofthemodel............................
2.5Multimodalinteraction..................................
2.5.1Integration....................................
2.5.2Pooling......................................
2.5.3Development...................................
3Auditoryprocessing:Exemplifyingoptimality
3.1Introduction........................................
3.2Archetypemodel.....................................
3.2.1Optimality:Anarchitecturalprinciple.....................
3.2.2Receptivefields:Acomputationalbuildingblock...............
3.3Technicalprospect....................................
3.4Biologicalprospect....................................
3.5Discussion.........................................
ix
vix112234568991113131314151818181920232323242525262627293132
Contents
35erceptionPLateral-line44.1Introduction........................................35
4.2Analyticdescriptionofshape..............................36
4.2.1Mathematicalformulation............................37
4.2.2Effectsofshape..................................37
4.3Hydrodynamicobjectrecognition............................40
4.3.1Definitionofhydrodynamicobjects.......................40
4.3.2Objectrecognitionperformance.........................42
4.4Discussionandoutlook..................................46
5InfraredSystemofSnakes49
5.1Introduction........................................49
5.2Warmingup:Signalmapping..............................51
5.2.1Athermodynamicvista.............................51
5.2.2Mappingtheheat................................57
5.3Sensoryencoding.....................................59
5.4Neuronaldecoding....................................63
5.4.1Minimalmodel..................................63
5.4.2Theextendedmodel...............................67
5.4.3Featureextraction................................69
5.5DiscussionandOutlook.................................73
6IntegratedMultimodalTeaching75
6.1Introduction........................................75
6.2IntegratedMultimodalTeaching(iMT).........................76
6.2.1HowdounisensorymapsdetermineiMT?...................77
6.2.2HowdoiMTcharacteristicsinfluencemapadaptation?............79
6.2.3HowdoesiMTcalibratedifferentunimodalmaps?..............84
6.3Modelresultsandexperimentalfindings........................84
6.3.1Experimentsprovision-guidedmapformation.................85
6.3.2Experimentscontravision-guidedmapformation...............87
6.4DiscussionandOutlook.................................87
AAppendix:OptimalMapformation95
A.1Arecipeformakingmaps................................95
A.2Nonlinearitiesininformationprocessing........................95
A.3Remainingderivationstepsleadingto(2.16)......................96
A.4Gaussianblurredsignal.................................98
A.5Pseudoinverse.......................................99
BAppendix:Lateral-linePerception101
B.1Planepotentialflow...................................101
B.2Rotational-symmetricpotentialflow..........................101
103iMTendix:AppCC.1Optimalcombinationoftwomodalities.........................103
C.2Optimalcombinationofthreemodalities........................104
C.3ModelParameters....................................105
x
bDieeginntReisezuvondeinentausendF¨ußen.Meilen
1.ProcessingSensoryInformation
Laotse
Thisthroughchapterthepglassesrovidesofthisawosynopsisrk,weofapplythewideOccam’sarearazoofrbysensorymeansinfoofintrormationducingproandcessing.foLocussingokingon
inthethesessentialubsequentconceptschapters.usedinItthisstartsthesiswithenabthelingusquestiontoofderivewhatparedictionsstimulusoninsensoryexternalperfoworldrmanceis,
discussestheconceptualstepsinthesensorypathway,andpresentsabriefoverviewofneuronal
repsynapticresentationplasticitbyy,ameanstoolofformaps.understaFinallynding,wefocusdevelopmourentofattentionaoncommonmapfomulti-sensormationryandspaceintrothatduce
enablesbeingstoactfaithfullyintheexternalworld.
1.1Stepsofsensoryprocessing
SensoryprocessingcanbedividedintothreeconceptualstepsasdepictedinFig.1.1(A−C).First,
astimstimulusulusinisaphspecificysicallywaymappandedontranslatetodetectorthephysicalorgans.signalSecond,intotheneuronaldetectorinforgansormation,respviz.,ondtospikthees
ofseethee.g.corr[164esp,267onding],andprimaryforthesensorydifferentneurons;levelsforofinmotrodelingductionstoneuroscienceneurons,e.g.neuronal[119].Third,computationthe
provideinformationausefulcontainedrepresenwithintationtheofspikthee-trainsoriginalofstimtheulussensoryto,forsysteminstance,isproinitiatecessedmotorneuronallycommands.soasto
Figure1.1:Threeconceptualsteps(A−C)ofsensoryprocessing.Everystimulusintheexternal
worldemitssignals,inourexamplelight.(A)First,thesignalsaremappedontothesensoryorgans
wheretheyaffectthephysicalstateofspecialized(protein)structureswithinthe(photo)receptors.
(B)Second,thesensoryneuronsrespondonlytospecificstimuli,forinstancebluelight,bymeans
ofgeneratingcorrespondingspikes.(C)Third,theneuronalsystemdecodestheimbeddeddata
withinthespike-trainstorevealtheessentialinformationinausefulway.
1
1.ProcessingSensoryInformation
mappingSignal1.1.1Initially,anyobjectintheexternalworldrevealsitspresencebygeneratingdifferentsignals.A
runninganimalmay,forinstance,generateasoundandachangingvisualimageasitmoves,as
wellasvibrationsandaninfraredprofile.Thegenerationandtransmissionprocessfollowsspecific
physicallaws.Thisfact,assimpleasitmaylookatfirstglance,providesapowerfultoolfor
understandingsensoryprocessing:themathematicaldescriptionofsensorystimuliasphysical
tities.quanableobservInconcreteterms,givenanysignalthatvariesxasafunctionofspatialpositionxandtimet,it
ispossibletocalculatethetime-dependentinputi(t)tothereceptororgans.Thatis,thephysical
mappingofsignalssx(t)ontosensorydetectorscanbedescribedbyasetoftransferfunctionsh,
h:sx(t)→ix(t).(1.1)
Moreover,realobjectsintheexternalworldtypicallyemitabundleofsignalsthroughseveral
channelsbasedondifferentphysicalpathwaysatthesametime.Thatis,theypermanentlyleavea
multi-sensoryfingerprintthatcanbedetectedandprocessedbysensorysystemstoformacoherent
representationoftheexternalworldinspace-time.Thisissuewillleadustomultimodalintegration
andlaterontothequestion,howthispreciseinterplayofoursensesevolves.Weleavetheissue
ofmultimodalprocessingforthetimebeingandnowresumediscussingthefundamentalstepsof
sensoryprocessingwithinonemodality.
dingencorySenso1.1.2Inthesecondstepofsensoryprocessing,thesignalthatismappedontothesensorydetectors
istranslatedintoneuronalinformation–spikes.Thisprocesscanalsobedescribedbyasetof
oftransferthefudetector.nctionsk.ComingInthebacbktoeginningtheofexampletheproincess,Fig.the1.1,externallightinsignalthevisualaffectstherangeph–ysicalsimplifiedstate
–changesthestateofthewavelength-sensitivephotoreceptorsintheretina.Bymeansofthis
physicalprocess,spikegenerationisinduced.Theresultingspikesti(t)consequentlyencodethe
intensityofthecorrespondingspecificwavelength[263,267,281]ataparticulardetectorpositioni
,ttimeandk:ix(t)→ti(t).(1.2)
issueTwisothataspectssensoryofsensoryencodingencodingincludesareatofleastfurthertwoinstepsterestofinsignthealconfiltetextring.ofthisThephthesis.ysicalThechangefirst
ofstatewithinthereceptorisspecifictoaparticularstimuluscharacteristicandhencefiltersthe
signal.Inourexample,thephotoreceptorrespondsonlytobluelight;forreviewofthevisualsignal
cascadeInseeaddition,[263].theSo,prothecesswavofspikelength-speecgenerationificityalsselectsofiltersoutthelightofsignal.allWotherenowcolors,takecf.aclFig.oser1.1.look
atthisimportantissue.First,translatingsensorysignalsintoneuronalinformationcomparesto
ananalog-digital-conversionprocessbecauseaspikecodeisabinarycodeconstitutedofzeros
(nospikesatthispointintime)andones(spikeatthisparticulartime).Second,intheprocess
oflimitspiktheesamplinggenerationrate,atthee.g.levelofrefractiononetimesinglelimitsneuron,thespikbiopheysicalfrequencydyn[am267ics,of164].cellThisrescompartmenultsintsa
restrainedinformationconsamplingtent.Third,accuracy,thethatneuronalis,thetransmissionanalog-digital-conprocessvalsoersionalteprorscesstheimbprincipallyeddedreduceinformation.sthe
Synapses,forinstance,typicallyfeaturecomplexnon-lineardynamicsthatcontributetoaniterative
sequenceoffragmentation,selection,andamplificationofinformation,andthusfilterthesignal
[89].information.Fourth,thereNeuronsarecandifferenresptondkindstoofspikeneuronalratesco(“ratedingcothatding”)butadditionallyinothermaycasesmodifyneuronstheresporiginalond
2
1.ProcessingSensoryInformation
tofiringonset,changeofspikerates,skipping,bursting,orcomplextemporalpattern–justtolist
afew[18,37,247].Wesee,generatingandtransmittingneuronalinformationpresentsdefinitely
afilteringprocess.Thereisnodoubt,however,thattheneuronalfilteringprocessesneednot
generallybebugs.Theymay,onthecontrary,benecessaryfeaturesreducingthearrivingphysical
signalstotheessentialonesandthusmakingexternalsensoryinformationstreamscomputationally
feasible.Thesecondimportantaspectofsensoryencodinginthisthesisistopography.Somesensory
organsareorganizedinsuchawaythattheyareabletoreceiveandconservetopologicallyordered
information.Inourexample,theretinashowsatopologicalorganizationofthedetectorsand
lensesassuringawelldefinedtopologicalrelationbetweenpositionsintheexternalworldxand
thedetectorpositionsi.Translatingspatialinformationfromtheexternalworldintoneuronal
information,however,inducesatransferfromcontinuoustopologiestodiscretetopologies.This
spatialanalog-digital-conversionprocessatleastcausesuncertaintiesinthesensorypathway.
Topography,moreover,isnotonlyaspatialcharacteristic.Itisalsoimportantinthetemporal
domain.Withintheauditorysystemitiscrucialtoknowthetemporalsequenceofeventsinorder
to,forinstance,processinterauraltimedifferences.Withinthelimitationofaneuronalsystemour
brainisevenabletotranslateneuronallytemporaldataintospatialinformationtogainauseful
representationofexternalspace.TheJeffress’modelisafamousexampleofsuchatranslation
[132].cesspro
dingdecoNeuronal1.1.3InTheythemanextycomestagefofromsensorydifferentprosenscessing,orytheneuronsineuronalwhichmasystemyhasadditionallytodecohadeveathespspikatialore-trainstempti(oralt).
relation.Inmathematicalterms,theneuronalsystemprincipallyhastosolvetheneuronalinverse
andproblem.signalThatmapping)is,theandsystemkhas(sensorytoinvertdetectingthecomandpositionneuronalofencotransferding)tofunctionsrevealhthe(signaloriginalgeneratingsignal
s.Ingeneral,itishardlypossibletofindanexactsolution.Thesystem,however,cantrytofind
anoptimalinversefunctionltoreconstructsˆtheoriginalsignalfromtheneuronaldata
l:ti(t)→ˆsx(t).(1.3)
Additionally,inmanycasesthesensorysystemevensettlesforcomputingtheessentialstimulus
informationinsteadofreconstructingthecompletestimulus.Energycostsandthelimitationsof
neuronalprocessingcapabilitiesmaybereasonsforthat.
Thelimitationseachsensorysystemisfacedwithleadtoanotherglobalissueinsensory
processing.Asensorysystemcanonlyposeandanswerthefollowingquestion:Givenuncertainties,
sensorylimitations,respandonse?aForsensoryillustration,response–thewhatfiniteisntheumbermostofprdeteobablectorsissignalnotthaablettocouldsamplehaveconcausedtinuousthis
signalsorspacewithoutuncertainties.Thelimitingtemporalaccuracyofneuronalcomputationas
wellmatterasthewhatneuronalcausesthesenoiselevelsystemadditionallyrestrictions,increasetheproquestioncessingitselfreduceimpreciseness.snaturallyConsequentoaqtlyues,andtionnoof
conditionalprobability.So,itisnotsurprisingthateverywhereinthecourseofsensoryprocessingwe
findneuronalstructuresbehavinglikeestimators[7,59,61,67,69,113,120,183,220,237,238,239]
that,forinstance,canoptimallysolvetheneuronalinverseproblem[75,76,252].Itremains,
however,torevealhowtheseestimatorsworkonthelevelofneuronalcircuitryandtoquantify
erformance.ptheir
3
1.ProcessingSensoryInformation
Figure1.2:Threecategoriesofmapsbasedontheirtopography;adaptedfrom[11,154,208].
NelsonandBower[208]suggestthreemapgroups,i.e.continuousmaps(A),patchymaps(B),
andscatteredmaps(C).(A)Mapcorrespondstoacontinuousmappingofauditoryspaceinthe
optictectumofthebarnowlshowingaglobaltopography[154].(B)Thepatchymappingofvisual,
auditory,andmultisensorysensitiveareasinthehumansuperiortemporalsculusmultisensoryarea
showsatopographyonlywithinsmallanddistinctareasofthewholeorganization[11].(C)Inspite
ofthespatialorganizationinthelattertwoexamples,thescatteredmappingofolfactoryinputto
thepiriformcortexoftheratlacksanyspatialorganization.Dependingonthecomputationaltask,
thedifferentstructuresleadtodifferentcomputationaladvantagesasbrieflyaddressedinthemain
text.
aimInofallsummarythe,theresultingphysicsneuronalofexternaleffortissignaltoachievtransdueactionreasonabledictateandtheusefulsensoryrepresedatantationstructure.ofThethe
originalstimulusinformation,viz.,thesignal.Thecentralnervoussystem(CNS)canrepresentthe
thecomputedtopologicallyinformationorderedindifferenneuronaltwamapys.Inplaythisanthesisessenthetialprole.opulationThecomapdeand,accuracymoredeimpfinedortanastltoy,
howpreciseamapcanrepresentanexternalstimulusisagoodmeasurefortheperformanceofthe
correspondingsensorysystem.Sowewillconsidermapsinthenextparagraphinmoredetail.
mapsNeuronal1.2Amajorroleinsensoryprocessingisreservedforneuronalmaps1[116,161].Theconceptofamap
suggeststhatthereisalawfulrelationbetweenthesurfaceoftheneuronalsubstrateandarelevant
aspectoftheexternalspace.Tocutalongstoryshort,aneuronalmapistypicallyanarrayof
neuronsinwhichneighboringneuronsrespondto“similar”sensorystimuli.Thissimplification
holdstrueatleastforspatialsensorymapsonwhichwefocusinthecourseofthisthesis.
Asanexample,visualinputinthemammalianbrainisprocessedthroughmultiplecorticallayers
thatareorganizedaccordingtothetopographyoftheretinalinputcells(“retinotopicorganization”)
[281,143].Inthisway,neighboringneuronsrespondtovisualinputfromneighboringpointsin
space,andthusformaspatialmap.Spatialmapshavebeendiscoveredinvarioussensorysystems
inmanygroupsofvertebrates[10,40,62,109,123,149,151,192,259,264,295]andeveninsome
[130].ertebratesnon-v1InReihenfolge1879,kHelmholtzennen[gelern114]tremarkwird,inedder“DasssichdurcihrehdasEindr¨Enucketlangf¨darbieten,uhrendesdassdiesetastendenReihenfolgeFingersansichdenalsObunabh¨jectenangigdie
davderenonEerwlemeneist,tenobmanmanmitimmerdiesemwiederodervor-jenemoderrFinger¨uckw¨artstastet,indassderselsiebenfernerOrdnnichungteinedurceinl¨hlaufenaufigmbuss,estimmumtevonReiheeinemist,
zumRiemann’sanderenTermizunoklogieommen,,einealsokeineMannigfaltigklinienf¨eitzwormigeeiterReihe,Ordnung,sonderndasalleseinfl¨istacleichenhahtfteseinzuseheNebn.”eneinander,Thatis,oderHelmholtznach
alreadyrecognizedthataneuronalrepresentationofatwo-dimensionalsurfaceconstitutesatwo-dimensional
brain.theinmanifold
4
1.ProcessingSensoryInformation
Figure1.3:Thefiringprofileofaspatialmapencodesthelikelihoodoffindinganobjectatacertain
positioninsensoryspace.Focusingonthefiringrateofonlyasingleneuronsuchasneuroni
withfaithfulppreferrederceptiondirectioncanonlyxibandeachievignoringedifitstheneighactivitbyorsofprevtheentswholeafaithfulneuronalpmaperception(justofcomparereality.theA
“neighborhood”ofiwiththerest)istakenintoaccount.Consequently,eventhoughthenominal
valueofthefiringrateofneuroniisidenticalincaseAandB,therepresentedphysicalreality
differssignificantlyinbothcases.
Neuronalmapsarenotlimitedtospatialrepresentations.Prominentexamplesarefoundin
theauditorysystem.Therearemapsrepresentingfrequency,interauraltimedifference,interaural
amplitudedifference,andevenamplitudemodulationfrequencies[84,187,214,225,245,265].So,
weseethatmapsareneitherrestrictedtoparticularsensorymodalitiesnorparticulardomains
space.ashsucAtypicalwayofcategorizinginphysicsistosearchforcharacterizinglengthscales.Lookingat
maps,wecanfindtwotypesoflengthscales.Firstandobviouslywecanconsiderthedimensions
oftheneuronalarray.FollowingthepresentationofDehaeneandCohen[58],onemayhence
distinguishmapsbytheirsizeintomacromaps(afewcentimeters),mesomaps(uptocentimeters),
andmicromaps(millimeters).Second,besidethelengthscaleofmapdimensionswecanconsider
thescaleuptowhichacommontopologicmaporganizationcanbeidentified.NelsonandBower
[208]thereforesuggestthreecategories,continuousmap,patchymap,andscatteredmap;see
Fig.1.2.Thesecategoriespresumablyarisemainlyfromcomputationalefficiencyreasonsaswewill
seeinthenextsection.Strictlyspeaking,thelatter,thescatteredmap,isnotamapasitshows
.yographtopnoIntrinsicallytiedtotheconceptofaneuronalmapisthatofreceptivefields.Areceptivefieldof
asensoryneuronisdefinedastheregionofspaceinwhichthepresenceofastimulusaltersthe
activityoftheneuron.Sincereceptivefieldsofsensorymapsareeasilyaccessibleexperimentally
theyarewellstudied,andthecorrespondingdatashedlightontheunderlyingmapproperties.
ertiesroppMap1.2.1Aswehaveseenjustabove,mapsareusedinmanyareasinthebrain.Thecompellingquestion
[116]thereforeis:Whatisthefunctionaladvantageofneuronalmaps?Thatis,whychooseamap
structureforneuronalprocessing?Here,wefocusonneuronalmapswithaglobaltopography,viz.,
continuousmaps;cf.Fig.1.2.Fromthetheoreticalperspective,thefundamentalargumentisthat,
incontrasttoarbitrarypopulationcodingorclusteredneuronalarchitectures[51],neuronalmaps
ensureatopographicneuronalorganization.
Thisconceptualadvantageofmapsarisesfromcombiningbasictopologicalinformationwith
actualstimulus-evokedresponses.Thatis,incaseofsensorymaps,thepreferreddirectionofthe
andsensoryactivityorgansalloorwsforreceptorsinisterpretinglinkedthewithfiringtheirprofileactualofmapactivity.neuronsInotherwithwrespords,ecttomergingtheirtopneighographbors,y
withviz.,conpreferredtext-depdirectionendent.xiFisortheexample,sameinas(A)illustratedand(B),ininFig.the1.3,contextalthoughofitstheneighbactivitoringyofneuronsneuronitsi
informationMoreover,conthetentmapdiffersorganizationdrasticallyfromunderlyingcase(Athe)to(Bneuronal).processingallowsforanefficient
5
1.ProcessingSensoryInformation
therepresenlikelihotationodinofterprconetintationuouslyofvaaryingfiringinputpatternsignonalsaasstospatialchasticmap,thatquantitiesis,.thFeorlikelihoinstance,odtoitallofindwsa
sensoryobjectatacertainposition[60,116,131,227,248].Inthecontextofmaps,wecannot
retrieveanythingfromthefiringprofileFig.1.3(B)butanoisybackgroundwhereasFig.1.3(A)
presentsquitenicelytheprobabilityoffindingacorrespondingsensoryobject.Thatis,thelatter
correlateswithapositionalestimatorforsensoryobjects.Whilesensorymapsencodingspatialor
temporalestimatorsarethemostevidentones,maps,especiallymapsinhigherbrainareas,are
notlimitedtospatialortemporaltopology.Theyarefoundtoencodevariousstimuliunderlying
vkindariousoftopmapisologiestopro(reviewvideabyKnsystematicudsenetal.represen[161]).tationSo,ofgettingcomplextostimthepuli.oint,therealpowerofany
sensoryIntheobconjecttextpresenoftedthisabthoevesis,isthecrucial.likelihoBeyodondinthisterpretationissue,ofmapstheprofiringvideotherpatternevokcomputationaledbya
advandanBowtages.er[208One,]assuggestedwehavethatseendifferentpreviously,categoriesrelates(seetotheFig.1.2)degreesofoftoptopologicologicorderorder.aredueNelsonto
minimizingcomputationaleffortsimilartoparallelcomputers.Thegeneralhypothesisisthatmap
overheadorganizationandanreflectsoptimalmapusageloadandbalancing.inputThestructureformerinorderandtotheachievlattereaareminimalfoundtocommbeunicationrealized
andneuronallyscatteredbythemap;tcf.hreeFig.kinds1.2.ofConmapstinmeuousnmtionedaps,abforove,instance,namely,areconfoundtinuoustobemap,usedpatchmainlyymap,for
localoperationsintheproblemspacesuchasfeatureextractionandspatialfiltering.Wehaveto
bearinmindthatcomputationaltasksareperformedmainlybutnotonlybymeansofsynaptic
connections.Chemicalprocessesarealsousedforinformationprocessing,forinstance,toinhibit
acompleteareaofneuronsrepresentingsimilarfeaturesofthestimulus.Thustheconceptofa
mapdoesnotonlyminimizeneuronalwiringeffortbutenablesreasonablechemicalcontrol.“Load
balancing”isachievedbylocalmapdistortionsthatreflectdifferencesinsensorydensities.For
instance,thereisamuchhigherdensityofsomatosensoryreceptorsinthetonguecomparedtothe
skincorrespinourondingback.sensoryThemapincreasedwheresensorymoreneuronssensibilityencoofdethethetonguesamepishysicalreflectedspacebythandistortionsinourofbacthek.
Thewithinpatcahymoremapsglobalseemcontotext.Butadditionallythisisincludestillnonmatter-loofcalopongoingerationsdebate.toTheanalyzeloscatteredcalsensorymapssucdatah
asidenfoundtifiesinthecomplexolstimfactoryuliandsystemanswreflectersthethequestionnon-topofographicwhatrathernatureofthanthethatinput.ofwhere.Here,thesystem
mapscomputationalofCollection1.2.2Thereisanevenmoreconvincingexplanationfortheomnipresenceofmapsinthebrain.The
realpowerofcomputationalmapscanbeappreciatedonlywhentheinterplayofseveralmaps
isarbitraryconsidered;transfforormationreviewinsee[the161].repreInsenthetationconoftextofinformation.thisthesisTowegiveadefinecfamousomputationalexampleofashoanw
informationisprocessedinasequenceofcomputationalmapsweconsidertheauditorysystem;cf.
ofFig.the1.7.Finferiorrequency-depcolliculusend(ICC).ingtimeInanextdifferencesstep,arethismapcomputediswithintranslatedainmaptoianthespatialcentralmapninucleusthe
theexternalopticnucleustectumof(forthereviewinferiorsee[collicul154,us155,(ICX)161]).whereAlthoughthethevinformationariousisearlyfurthersensoryprocuesjectedinintheto
auditorysystemarehardlyrelatedtospace,intheoptictectum(OT)theauditorymaprepresents
spaceandevenconnectssmoothlytospatialmapsofothermodalities.
beautifulTheOTexampleanditsofwtheell-studiedconnectionbmammalianetweendifferenhomologue,tmaps.thesupAlleriorsensorycolliculussystems(SC)that[261pro],isvidea
minformationultisensoryasinwaellasmap-likepredominanformprotlyjecttomonosensorythelaSC/OTyers[are261];foundcf.andFig.are1.4.shownWithintoproctheessSC/OT,spatial
6
1.ProcessingSensoryInformation
Figure1.4:Thesuperiorcolliculus(SC).Thesuperiorcolliculusanditsnon-mammalianhomologue,
thethatproopticvidetectum(informationOT),areinwaellmap-likstudeiedform[mid-brain261].stWithinructuresthetoSC/OT,whichallspatialsensorysensorysystemsinformationproject
ofinformationdifferentisusedmonosensoryfordelivmapseringisunamcomputedbiguousandcommotorbinedincommands,tomforultisensoryrefiningmaps.sensoryThemocomdalities,bined
andSC/OTforarefurthermutuallyprocessingaligned,inwehighercanspbraineakofareasa[166collection,167,of179maps,216that,260].represenBecausetanallalignedmapsinsensorythe
space.Thefigureisadaptedfrom[179].
information.Inmultisensorymaps,mostneuronsrespondtomorethanonesensorymodality.In
doingso,theneuronalresponsetomorethanonesensorymodalitycanbecomplex.Incaseofspatial
andtemporalcoincidencewecanobservesuper-additivity,cross-modalenhancement[134,224,261],
orinverseeffectiveness[224]whereasincaseofnon-coincidencecross-modalsuppressioncanbe
found[140].Multimodalmapsarewidespreadinthebrain[139].
BoththemultisensorymapsandthemonosensorylayersarefoundtobealignedintheSC/OT
togainaunifiedmultisensoryrepresentationofsensoryspace[147,261].Thecombinedsensory
informationcanthenbeusedtogeneratedirectionalmotorresponses[166,167,179,216,260].This
isonlypossiblesince,withintheSC,therearenotonlysensorybutalsomotormaps,i.e.,motor
neuronsorganizedinamap-likestructure.Inmotormaps,systematicvariationsofamovement
arerepresentedtopographicallyacrosstheneuronalarray.Directevidenceforthishypothesishas
recentlybeenfoundineye-trackingexperiments[104].
Inacollectionofalignedneuronalmaps,moreover,externalobjectscanbeidentifiedbytheir
positionencodedbymeansofthefiringpatternindifferentsensorymaps.Thisdoesnotmeanthat
anobjectisadequatelydescribedifonlythepositionisknown.Itratherimpliesthatpositionserves
asfordefiningappropriateaandsensory–inobconject;trastcf.toFig.1.5.higherWhenccomputationalombininglevelsdifferen[65t,98]sensory–necessarysystems,theinformationspatial
informationisneededtobindinformationassociatedwiththesamepositioninthemonosensory
mapsintoonesinglemultimodalperceptforfurtherprocessing.
7
1.ProcessingSensoryInformation
Figure1.5:Objectformationbyspatialinformationbinding.Withinanatlasthereisasimple
waactivitytoydefinepatterna(redsensoryobcolumn).jectbyUptomeansnoofw,sucgroupinghamecspatialhanismisinformation,speculativviz.,e.Hospatialwever,congrueitcannt
bedynamicrelevanent,forvironment.instance,Theretoareselectlinesamofultimoevidencedalsupptargetorting(e.g.therunningexistenceandofsucbarkinghamecdog)hanism.withinAa
andprominenvisualtmapexample[210].aretheTheseneuronalcircuitsare“AND”suppandosedto“OR”idencircuitstifyexternalwithinobthejectsOTsucthathaslinkpreyinfraredand
enhancetheircontext-dependendsaliency.
Bearinginmindtheadvantagesofneuronalmapsthatareduetotopographywenextfacethe
questionofhowaneuronalmapcandevelop.Laterinthisthesis,weadditionallystudyhowa
collectionofmapscanevolve.
developmentMap1.2.3Onemightarguethatneuronalmapssimplyexistbecausetheirneuronalarchitecturefollowsthe
sensorysurfaceoftheirinputmodality.Fromthispointofview,forinstance,thevisuallayersare
retinotopicallyorganizedbecausetheyreceivetheirinputfromtheretina.Similarly,afrequency
mapjustreflectsthetonotopicorganizationofthecochlea.
Interestingly,thisargumentdoesnotholdforeverysensorymapasitalreadybecomesapparent
intheexistenceofauditorymapsforinterauraltimeandamplitudedifferences.Regardingexamples
inthespatialdomain,letusconsiderthesensoryunitsofthefroglateral-linesystem[295]orthe
snakeinfraredsystem[109].Thesesensoryunitsreceiveacomplexsuperpositionofinputfrom
severaldifferentspatiallocations.Computingamapfromsuchacomplexinput[75,76,95,252]is
certainlynotstraightforward.So,howcanmapsevolve?
Themostobviousideaistoimagineanexplicitgeneticmechanismencodingmaptopography,
viz.,connectivity.Inthiscase,however,weha12vetogenetically15encodealargenumberofpossible
connections.Ourbrainconsistsofabout10neuronsand10synapticconnections.Thatis,
wehavetoencodeanumberofpossibleconnectionscomparabletothenumberofatomsinthe
universe[267].Ofcourse,our2−3∙104humangenescannotachievethisdirectly.Consequently,
intrinsicmechanismsmustdeterminemapdevelopment.
Inthelastyears,neurobiologistsformulatedfourmainhypothesesastohowtopographic-correct
synapticmapconnectionsarise;reviewse.g.[34,181,190,271].First,followingtheaddress
labelhypothesis,synapticstructurespossessaseriesofpositionallabelstellingingrowingaxons
orpre-synapticaxonterminalshowtoarrange.Second,neighboringaxonsconservepre-synaptic
topographyduringthegrowthprocess.Therefore,thepost-synapticstructuresaretopographically
arranged.Third,boththeingrowingaxonsandthetargetstructuresdevelopinatopographically
8
1.ProcessingSensoryInformation
arrangedtimesequence.Thetemporallymatchingentailsaspatialmaptopography.Fourth,pre-
synapticaxonterminalsarrangethemselvestopologicallybymeansofelectricalactivity.Although
evidenceforeachofthesehypothesescanbefound[271],severalmechanismsseemtocooperate
toensureapropermaptopography.Togiveanexample,electricalactivity-basedadaptionis
supposedtofine-tunemapconnectivityaftersomeimpreciseprewiringwhichcanbebasedonother
mechanismssuchaschemicallabeling[13,146].
Precisemaptopographyiscrucialforsensoryperformance.Inthenextsection,wetherefore
shedlightonactivity-based,viz.,spike-dependentmapformationbyconsideringlocalmechanisms
andglobalstructuresresponsiblefortheplasticityofmaptopography.
1.3Learninginsensorysystems
InThattheis,consensorytextofactivitsensoryysys(input)tems,leadslearningtoadifferenbasicallytmapmeansactivitacyhange(output).intheinpuConsequent-outputtly,learningrelation.
tak1949,esplaceHebbbyalreadymeansofctranslatedhangingthisthegeneralconnectioninsightofstrengthsactivitbetwy-basedeen(inputplasticitandyintooutput)neuronalneurons.termsIn
byformulatinghisfamousrule2[112].Inshort,ifapost-synapticneuronexcitesaspikeshortly
wafterords,oneHebb’sofitslearningpre-synapticruleneudescribronseshasthatfired,tempthisoralparticularcorrelationsofconnectionneuronalisstrenactivitygthened.–ifIncausalother–
wareorldlearned.andthTusocutHebb’salongrulestoryimpressivshort,elyneuronalenablesusactivittoyrelatesunderstandtypicallyverytoevgenerallyentsinhothewtoexternallearn
correlationintheexternalworldbymeansofalocalneuronalmechanism.
withIntheHebb’sconruletextofarise.thisFirst,thesiscanw“PredictingeSeunderstandnsoryPaspectserformance”oftsensorywopquestionserformancdirectlyebymeansconnectedof
studyinglocalmechanismsofsynapticplasticity?Second,whatprovokeslearning,i.e,theadaption
ofsensoryabilities?Moreprecise,whichglobalandintrinsicmechanismsassureproperguiding
inputalignedforlocollectioncalofsynapticmapsplasticitfoundyintotheevolvSC?eacomplexbrainstructure,forinstance,suchasthe
mechanismscalLo1.3.1Inthelastdecades,neurobiologicalevidencehasbeenfoundtosupportHebb’srule.Various
mechanismsunderlyinglong-termpotentiation(LTP)andlong-termdepression(LTD)ofsynaptic
transmissionhavebeenstudiedingreatdetailevenifnotyetfullyunderstood;reviewse.g.
185].177,128,34,[13,Inaddition,togainfurtherinsightintothistopic,Hebb’sessence,synapticweightdevelopment
duetocorrelatedactivityofcorrespondingpre-andpost-synapticneurons,hasalsobeenformulated
inmathematicalterms;fordetailedreviewseee.g.[115,145].Thedevelopmentofsynaptic
connectionstrengthJijbetweenapre-synapticneuroniandapost-synapticneuronjcanbe
dividedintothreepartsasillustratedinFig.1.6.First,thearrivalofapre-synapticspikeattime
timinducestheweightJijtochangebyanamountηwpre.Second,iftheconsideredpost-synaptic
neuronjexcitesapost-synapticspikeattnj,allconnectionsarechangedbytheamountηwpost.The
quantityη>0isasmallparameterandtheweightchangeswpreandwpostcaneitherbepositiveor
negative.InFig.1.6weassumedwpre>0andwpost<0.Thethirdpartisduetothespike-timing
ofpreandpostspikes.Giventhetimedifferencestim−tnjbetweenpre-andpost-synapticspikes,
JijischangedbytheamountofηW(tim−tnj),wherethelearningwindowW(tim−tnj)denotesa
2gro“WhenwthanproaxoncessoformcelletaAbisolicnearchangeenoughtaktoesexciteplaceincelloBneandorbrepotheatedlycellssucorhperthatsistenA’stlytakefficiencyes,partasinonefiringoftheit,somecells
firingB,isincreased”;[112]p.62
9
1.ProcessingSensoryInformation
Figure1.6:Thesynapticlearningprocess[115,145,146].Thediagramillustratesthelearning
processduetothethreepartsofSTDP.Inthiscase,everyspikeofthepre-synapticspiketrain
(toprow)strengthensthesynapticweightJijbywpre(green).Allpost-synapticspikes(second
upperrow)induceanegativeweightchangewpost(orange).Thecontributionofthethirdpart
ofSpike-timing-dependentplasticity(STDP),thecorrelationterm,dependsonthespiketiming
andthelearningwindow(middlerow).Intheleftpart,thepairofpre-postspikesistemporally
toofarseparatedascomparedtothedimensionofthelearningwindow.Theremainingtwoother
possibilitiesaredepictedintherightpart.Ifapre-synapticspikeprecedesapost-synapticspike,
thesynapsestrengthincreases(blue).Incaseofoppositetemporalorder,thesynapseisweakened
(purple).
ofrealFig.valued1.6.Finallyfunction.,weWecanhavecombinedepictedtheatthreeypicalpartsformandofaderivelearningaHebb-twindoypweinthelearningthirdruleuppbasederroonw
spike-timingforatimeinterval[t,t+Δt]
ΔJij=ηwpre+wpost+W(tim−tnj).(1.4)
timtnjtim,tnj
npTheabost-synapticoveequationspikecantrainsaltsopre;pbeost(t)form=ulatedδ(tin−intin;j)tegralandform.computeWethereforedefinethepre-and
ΔJij=ηt+δtdtwpretipre(t)+wposttjpost(t)+t+δtdtW(t−t)tipre(t)tjpost(t).
tt(1.5)Both,correlateequatipre-onand(1.4)pandost-synaptic(1.5),represenactivittya.Inmathematicalparticular,theHebb-tlearypeninglearningruleisrulebasedsinceonthetheequationsprecise
timingsequenceofpre-andpost-synapticspikes.Suchlearningschemesarethereforedenotedas
Spike-timing-dependentplasticity(STDP);fordetailedreviewseee.g.[1,15,55,78,115,145].
Kempter,preGerstner,andvanpHemmenostdemonstrated[145]thatlearningschemesbasedon
pre-synapticνiandpost-synapticνimeanfiringratessuchas
ddtJij=a0+a1νipre+a2νjpost+a3νipreνjpost+O[(νipre)2]+O[(νipost)2](1.6)
arenotcapabletocatchtheneuronaldynamicsonaplausibletemporallevel.Togiveanexample,
meanratedescriptionsrequirethatallchangesofneuronalactivityareslowatatimescaleofthe
learningwindowW.Thatisnotnecessarilythecase.TheHebbianlearningwindowincortical
areas,forinstance,isabout100msascomparedtothefrequencyof40Hzcorrespondingtoatime
scaleof25msonwhichneuronalactivityvaries.
10
1.ProcessingSensoryInformation
Figure1.7:Siteforplasticityandoriginofguidinginputformapadaptionintheauditorypathway;
adaptedfrom[78,156].Theauditorysystemprocessestheinitialsensorycues,forinstance,interaural
timedifferences(ITD)togainaspatialrepresentationofsoundsourcelocationsintheoptictectum
(OT)thatisalignedtomultimodalmapsintheOT.Inshort,sensorycues,i.e.,frequency-specific
ITDmaps,arepresentinthecentralnucleusoftheinferiorcolliculus(ICC)andareprojected
totheexternalnucleusoftheinferiorcolliculus(ICX).Fromthere,spatial-auditoryinformation
ismappedtotheoptictectum.Experimentsprovidingabnormalvisualinputdemonstratetwo
importantfindings.First,thereisaspecificsiteforplasticity.Thesynapticconnectionsbetween
ICCandICXchangeinsuchawaythattheICXmapacquiresOTtopography.Second,audition
adaptstoinputfromtheOT.Inresult,visionseemstoguideauditorymapdevelopment[159,160].
ThisshortexampleillustratesnotonlytheadvantagesofSTDPascomparedtorate-based
learningschemesbutalsoreflectsanissuealreadymentionedabove.Thewayofneuronalencoding,
forinstanceratecodingversusspikepatternsthatarephase-lockedtotheinputsignals,mayreduce
andrefinetheimbeddedinformationandthuscorrespondtoaneuronalfilterintheprocessof
neuronalencoding;seeSec.1.1.2.
Inthissectionwehavequantitativelyseenthatpre-synapticinputthatdrivespost-synaptic
activitycaninduceplasticityofcorrespondingconnectionstrengths.Tounderstandandpredict
theconsequencesoflocalsynapticeventstosensoryperformancewehavetoconsidertheglobal
contextwhereinthelocaleventstakeplace.
1.3.2Globalstructures
Inthissection,weintroducesome,butnotmanyimportantthoughtsaboutlearningbeyond
localsynapticevents.Locallearningrulescandescribehowthelocalconnectionstrengthchanges
accordingtothecorrespondingpre-andpost-synapticactivity.However,withoutknowingwhat
“corresponding”meansinthecontextofaspecificneuronalsetupofasensorysystem,locallearning
rulescannottellushowthesensorysystemdevelopsasawhole.Therefore,wehavetoaskatleast
threequestionsthatwillguideusthroughthesubsequentlines.First,whatdrivespost-synaptic
activitythatresultsinsynapticplasticity?Second,whereinthestructureofpre-andpost-synaptic
synapticconnectionsactivitdoyestoresplasticitultyinappreasonableear?Third,globalhowconnecdotiesons?theHereunpre-synapticto,wetakinputeanmanipulateexemplaryploost-ok
attheauditorypathway.
Wehaveseenabovethatpre-sypaticinputdrivingpost-synapticactivitycanleadtochangesof
thedrivescorrespthepondingost-synapticconnectionactivity?strength.Thatis,Thewhicfirsthinpquestionutprobeyjectsondtolothecalpossynaptict-synaptevicentsneuron.is:WhatTo
11
1.ProcessingSensoryInformation
giveaconcreteexample,aswecanseeinFig.1.7theexternalnucleusoftheinferiorcolliculus
(ICX)receivessensoryinputfromthecentralnucleusoftheinferiorcolliculus(ICC)andtheoptic
tectum(OT).Consequently,theactivityofneuronsinthespatialmapinICXisdrivenbytwo
inputorigins.Verycloselyconnectedtothefirstquestionisanotherone:Whatconstitutesthe
inputfromOT?AsthisistopicofChap.6,weleaveitforthemoment.
Topredicthowtheauditorysystembehaves,wehavetoconsiderthenextquestionofwhere
plasticityappears,i.e.,whichconnectionsarechangedbymeansofactivity-basedlearning?We
knowthatmapsintheOTaremutuallyalignedandthatthisisaresultofsensoryexperience
duringgrowth.Theanswertothewherequestioncanhencetellus,forinstance,whichinputguides
alignmentandwhichconnectionsadapt.Inourparticularexample,weseethatonlytheICC–ICX
connectionsshowplasticityandadapttoinputfromOT.Thelatterinputisthereforedenotedas
guiding;cf.Fig.1.7.ForadetaileddescriptionoftheICC-ICXconnections,considerKnudsenet
[156].erpapal.’sHavingclarifiedthewhatandwhere,“only”thehowremains.Incaseofourexample,Friedel
andvanHemmen[78]havemathematicallyanalyzeddifferentkindsofguidinginputandtheir
practicalconsequencestothelearningprocess.Theyderived,forinstance,thatonlyinhibitory
inputfromtheOT,calledselectivedisinhibition,canreorganizemaptopography,viz.,theICC–ICX
mappingcanbereorganizedsoastocompensateevenaspatial180◦shiftbetweenguiding(OT)
andsensory(ICC)input.ThiscorrespondstofamousprismglassexperimentsdonebyKnudsen
andothersinthebarnowl[156,159]–performedsimilarlyinotheranimals,suchashamster
[200],cat[278,280],clawedfrog[44],ferret[148,150],andsnake[96].Incontrast,excitatoryinput
leadstoalessprecise,lessstable,andslowerlearningprocesswhichresultsadditionallyinan
ambiguousmapping[78].However,inotherscenariosexcitatoryguidinginputfaithfullydescribes
174].[146,yrealitneurobiological
Insummary,derivingmathematicaldescriptionsoflocalsynapticevents,thatis,makingthem
quantitativelypredictive,isimportantbutnotenoughtogaininsightintothedevelopmentof
brainstructuresandtheirfunctionasawhole.Wehavetoconsidertheglobalenvironmentin
enwhichvironmenthetissynapticevconstitutedentsbappyearmeanstoofpredictneuronalwhatmapstheythatcause.bInelongthetoconsensorytextofsthisystems.thesis,Wesucprohvidane
anewmodelforintrinsicmapdevelopmentinthelastchapterofthiswork.Inthenextchapter,
wewillfocusourattentiononageneralframeworkofoptimalmapformationtounderstandhow
sensoryinformationcanbeprocessedbymeansofmapsandhowtheneuronalconnectivitypattern
predicted.ebcan
12
2.OptimalMapFormation
Mansollteallessoeinfachwiem¨oglichsehen-
einfacher.nichtaucherabertAlbEinstein
Inthestruggleforsurvivalinacomplexanddynamicenvironment,naturehasdevelopeda
multitudeofsophisticatedsensorysystems.Inordertoexploittheinformationprovidedby
thesesensorysystems,highervertebratesreconstructthespatio-temporalenvironmentfromtheir
sensorymodalities.Asweintroducedinthelastchapter,mostanimalsareabletocompute
aneuronalrepresentationoftheoutsideworldinformofaneuronalmap.Herewepresenta
universalframeworkthatallowstocalculatethespecificlayoutoftheinvolvedneuronalnetwork
bymeansofageneralmathematicalprinciple,viz.,stochasticoptimality.Thatis,givenaknown
physicalsignaltransmissionandrudimentalknowledgeofthedetectionprocess,ourapproach
allowstoestimatethepossibleperformanceandtopredictneuronalpropertiesofbiological
systems.rysenso
ointprtingSta2.1Amousehearsarustlinginthegrass,seessomeleavesmovingand–escapesfromthepredator.
Thus,theperceptionoftheoutsideworldbysensorysystemsandtheconsequenttranslationof
theirresponseintoareliableneuronalrepresentationthatallows,forinstance,directionalmotor
commandsisanessentialconceptforsurviving.
Inthelastchapter,weintroducedtheadvantageousconceptofaneuronalmapforrepresenting
theexternalworld(seeSec.1.2),forinstance,theauditoryspaceintheexamplejustmentioned
intheaboveparagraph.Moreover,wesubdividedthesensoryinformationprocessingintothree
fundamentalsteps.Thesearesignalmapping,sensoryencoding,andneuronaldecoding;seeSec.1.1
andtheexampleinFig.1.1.Inthischapter,wewillnowtranslatetheseconsiderationsintoa
mathematicalframeworkofoptimalmapformation.
delmoMathematical2.2Followingtheideaofmapping,encoding,anddecoding,wewanttounderstandhowtoextract
thestimulusatthebestfromthesensoryresponses.Mathematicallyspeakingwehavetoderive
theinversetransferfunctionl(1.3)thatcanperformanoptimalreconstructionofaparticular
stimulusgiventhesensoryresponsesandcomputationallimitations.Thisinversetransferfunction
canthenbetranslatedintoaneuronalconnectivitypattern(seeSec.1.1).
Thederivationweprovidebelowisbasedonthreereasonablesimplifications.First,weassume
thatallsensorymapsarepurelymonosensory.Althoughithasbeenquestionedwhetherthereare
mapswithoutinfluencefromothersensorysystems[262],ourassumptioncanbejustifiedbythe
findingthatmanyspatialmapsareclearlydominatedbyasinglesensorymodality[280].
13
OptimalMapFormation
receptorSecond,inputwewithsimplifyitstherespinonse.ternalThatprois,cesseswesetwithintheasptransferecificsensoryfunctionkreceptor(1.2)btoybeidenthetifyingidentitthey
connectsfunction,toandthehencecorresphondi(1.1)ngconnectssensorydrespirectlyonsestheonastimulusone-to-onecbasis.haracteristics,Althoughi.e.,thethismasensoryynotinputbe
givenforarealisticsetting,theframeworkofoptimalmapformationiswithoutlossofgenerality
btheecausefirstwetransfercanincorpfunctionorateh.theInreturn,filteringwepropgetaertiessetofoftheequationssecondthattransferaremorefunctionkbunderstandableyredefiningand
comparablewithcommontechniques;seee.g.Chap.5.
respTheonseofthirdtheimportansensorytsystem.assumptionThatisthatmeansofalinethatartherelationdetecbtoretwrespeentheonsesstimchangeulusandproptheortionallyreceptor
tothesignalstrength.Forexample,themultipolesofasubmergedmovingobject,i.e.the
signalstrengths,translatelinearlytothedetectablewatervelocitiesatthelateral-lineorgans
[[169251,,75213].,172,Nonlinear136,53],relationcansinbetwprincipleeenstimbeulustreatedandwithdetectorourrespmodelonses,aswe.g.,ell;aseelogariAppthmicendixrespA.2onsefor
details.
2.2.1Definitionoftheproblem
Aswehaveintroduced,anobjectgeneratesasignalsx(t)varyingintimetandpositionxinthe
externalworld.Thesignalmaybe,forinstance,thetime-dependentsoundpressureataparticular
locationormaydenotethepresenceofedgesormovementataparticularpositionwithinthevisual
field.Thesignalinducesaresponseri(t)inasetofNsensorydetectors.Dependingontheproblem
atorparthandofaasingledetectordetectorarrayisucwithhas0a≤spi≤ecificNincantervbaleaofbestcompletefrequsensoryrnciesinorgan,thecocsuchhlea.asIntheleftprinciple,ear,
thedetectorcombinesinformationfrompastsignalswithinthewholesensoryspace.Theresponse
isthereforedescribedby
tri(t)=dxdτsx(τ)hix(t−τ)(2.1)
spaceall−∞xThewheretransferthetrfansferunctionfcanunctionbehi(different)tincorpforeacorateshthedetectorphi.ysicsofAuditorysignaltransfertransmissionfunctions,andfordetection.example,
differincorpbetorateweentherighptositionandleftofear.soundInsourcegeneral,andweearcanwithsafelyrespeassumecttothatthehixhead(t)=0midlineforandlargevaluesthereforeof
|xstate|andofta.Thissensor.reflectsWewillourinneedtuitionthispropthatertevyentslateroccuon.ringMoreofaravwer,ayorsincelonganyagodetectorwillnotcaninfluenceonlyreactthe
xtofunctiontemp(2.1)oral-causal,withi.e.,adaptedpastinsignalstegrationwesetlimitshi(tas)a=0conforvtolution<0.Wwithercanespthenecttorewritetime,theresponse
ri(t)=dx∞dτsx(τ)hix(t−τ)=:dx(sxhix)(t).(2.2)
−∞theTheabodetectorverespequationonseisdescriblimitedesthebyrespatonseleastofthreeanidealfactors.system.Inbiologicalsystemsthequalityof
system.First,Second,informationnoisemayinfluencesgetlostalldurinstepsginthethetransferdetectionfromandtheoutsidereconstructionobjecttoprothecess[inside71].Finalsensoryly,
limitationsoftheneuronalhardware,forinstance,thelimiteddynamicrangeofreceptors,constrain
possiblesolutions;seeSec.2.4fordetails.
additionalWithinournoiseterms.mathematicalAccordinglymodel,waetermincorpdesoratecribingthesebacthreekgroundrestrictivnoiseeξx(tfactors)mbustybinetroaddedducintog
14
FMapOptimalrmationo
Figure2.1:Physicalmapping:signalsx(t)withbackgroundnoiseξx(t)ismappedontoanoisy
receptorresponseri(t)+χi(t)throughthenoisytransferfunctionhix(t)+ηix(t).Optimalmap
formation:the(possiblynoisy)inversetransferfunctionlix(t)+λix(t)givesanestimatesˆx(t)ofthe
signal.Signalsx(t)+ξx(t)
Transferfunctionhix(t)+ηix(t)
Receptorresponseri(t)+χi(t)
Inversetransferfunctionlix(t)+λix(t)
Estimatedsignalsˆx(t)
Table2.1:Functionsanderrortermsdescribingdetectionandprocessingofsensoryinformation.
theadditionalsignal.Fnoiseteurthermore,rmsηix(wt)eandassumeχi(t),thatresptransferectively.functionConsequeandntly,sensory(2.2)isrespmoonsedifiedaresohampastoeredreadby
ri(t)=dx[(sx+ξx)(hix+ηix)](t)+χi(t).(2.3)
Toreconstructtheestimatedsignalfromthedetectorresponsesri(t),theabovetransformation
musttransferbe“invfunctionsertedl”x(int)bsomeetweenappropridetectorateiwayand.Wtheethemaprefoatrepositcalculateionx.theWhentime-depapplyingendlexn(tt)intoversethe
iireceptorresponsesati,weobtainaccordinglyto(1.3)theestimate
sˆx(t)=[ri(lix+λix)](t)(2.4)
ioftheoriginalsignalsx(t).Herethehatonsˆx(t)denotesareconstructionandthetermλix(t)
noterepresenthattsintheconnoisetrastduetoeltostheewhere[concrete217,229]realizationthepreofsentthemodeltheoreticalisinvnon-iterativersee.transferThiswillfunction.resultWine
apurelyfeedforwardnetworkstructurewhenitcomestoaneuronalrealizationinSec.2.4.
Figure2.1illustratesthewholemathematicalprocedureofsensoryinformationprocessingas
describsection,edweinwillthesteindicatepsofhoFig.wto1.1.Allcalculatetheinvreleverseanttransfertermsaresfunctionsumlixmarized(t)thatinTenableable2.1.optimalInthesignalnext
reconstruction.2.2.2Optimalreconstruction
Wewanttotuneoursensorysystemtooptimallyreconstructnotonlyonespecificsituationbut
thethatwtypicealdenoteenasvironmen“tt.ypical”.InotherwConsequenords,tly,abiologicallyspecificrelevsensoryantsignsignalsalisbaelongconcretetoaclassofrealizationsignalsof
aminimizeclassofttheypical,expectationbiologicallyvalueofrelevtheantsquaredsignals.Thatdifferenceis,bitetiswaeenstocsignalhasticandquanretity.constructionWe.therefore
ofphThisysicalispossiblemappingbandecausetheallneuronalquantitiesprocesandsoffunctionsoptimal(seemapFig.formation2.1)inv(Secolv.ed2.4)inbareoth,self-athevproeragingcess
aswedemonstratein[35].Themathematicaldefinitionofself-averagingallowsforadescriptionin
termsofexpectationvalues.
15
OptimalMapFormation
Toderivetheinversetransferfunctionslix(t)thatenableoptimalsignalreconstructionfora
classoftypicalsignals,wecannextminimizetheexpectationvalueofthesquarederrorbetween
signalrealandestimatedE{lx(t),t}:=dtdx[sx(t)−sˆx(t)]2=dtdx[sx(t)−sˆx(t)]2.(2.5)
tt
t−Tt−T
Herethebrackets.denotetheexpectationvaluewithrespecttothedifferenttypesofnoise,and
Tisatypicalprocessingtime.
Tobemathematicallyprecise,anexpectationvalueisanintegralonaprobabilityspacewith
respecttoaprobabilitymeasurep.Forarbitraryfunctionsfandg,if|f−g|2=0thenf=g
withrespecttopor,physically,lookingattheworldthroughp’sglasses:whatpfindsimportant
popsupclearlywhereaswhatpfinds“irrelevant”hashardlyanyweight.Thelatterneednot
correspondtowhatwe“think”ourselves;seevanderWaerden[276].
Aquadraticformoftheerrortermhasbeenproventobeareasonableandpracticalchoicein
manyphysicaloptimizingproblems;see[193].IncaseofindependentGaussianerrorterms,the
formulationviaaquadraticerrorisundercertainconditionsidenticaltoresultsobtainedbymeans
ofmaximum-likelihoodestimates[135,144];seeSec.2.3.2.
Mathematically,theerror(2.5)isafunctionalassigningtoeverysetofinversetransferfunctions
onespecificvalue.Minimizationoffunctionalsintheaboveintegralformisacentralandwell-
studiedaspectofthecalculusofvariations[45,85,138,32].Forthepresentsituation,thefirst
variationwithrespecttoeveryinversetransferfunctionlj(x,t)istovanish.Thatis,
∂[sx(t)−sˆx(t)]2
∂ljx(t)=0foreveryj.(2.6)
Inordertosolve(2.6),wehavetosubstitute(2.4)fortheestimatesˆx(t)andreplaceri(t)byits
description(2.3).Expandingthesquare,weencounterexpectationvaluesofproductsconsistingof
varyingcombinationsofnoiseandsignalterms.Hereweassumethatallnoisetermsaswellasthe
signalitselfarestochasticallyindependentofeachothersothattheexpectationofaproductof
independenttermfactorizes;forinstance,
sx(t)ηix(t)=sx(t)ηix(t).(2.7)
Foraproductconsistingofthexsamekindoftermweneedtoconsiderthedefinitionofthe
autocorrelationofaquantityf(t)asgivenby
fx(t)fx(t)=δ(x−x)δ(t−t)(µf2+σf2)(2.8)
withµfthemeanandσfthevarianceofthequantityfx(t).Thatis,weassumeinafirststepthat
thevaluesfordifferentspatio-temporalpositionsarecompletelyuncorrelated,akindofworst-case
analysis.Sincethemeansofallnoisetermsµfvanishwegetthefollowingcorrelationterms
ξx(t)ξx(t)=δ(x−x)δ(t−t)σξ2,(2.9a)
χi(t)χj(t)=δijδ(t−t)σχ2,(2.9b)
ηix(t)ηjx(t)=δijδ(x−x)δ(t−t)ση2(2.9c)
with|x|<xmaxand0<t<tmax.(2.9d)
16
rmationoFMapOptimal
Throughthefinalequationwetakeintoaccountthatthenoiseηix(t)vanishesforlargevaluesoft
and|x|,inthesamewayasforthetransferfunctionhix(t).
Theautocorrelation(2.8)ofthesignalsx(t)itselfdependsontheproblemathand.Eitherthe
detectorsofthesensorysystemmeasureabsolutesignalstrengths(µS),e.g.,vision,ormodulations
ofameanvalueofthesignal(deviationσS),e.g.,audition.Inanycase,onehastochoosethe
correspondingbiologicallyrelevanttermandputtheotherequaltozero.Inthefollowing,we
choosetheexpectationvalueµS2ofthesignalastheappropriatequantityandthereforetakeσS2to
zero,ebsx(t)sx(t)=δ(x−x)δ(t−t)µs2.(2.10)
While(2.9)incorporatesreasonableassumptionsforallnoiseterms,thecorrelation(2.10)forthe
signalisastronghypothesis.Signalsarenamelycharacterizedbyspatio-temporalcontinuity;e.g.,
objectsandtheircorrespondingsignalsusuallydonotdisappearfromonepointintimetothenext.
termcorrelationGaussianAsx(t)sx(t)=Aexp−|x−x|2/(2σx2)exp−|t−t|2/(2σt2),(2.11)
forinstance,cantakeintoaccountcorrelationsbetweenneighboringpointsinspaceandtime.Here
σxandσtaretypicalspatialandtemporalcorrelationscales.TheapplicationofsuchaGaussian
correlation,however,doesnotgreatlyalterthefurtherderivationbutonlysmoothensthefinal
estimatedsignal.Forreasonsofclarity,wewillthereforesticktotherelation(2.10);seeAppendix
details.for[35]andA.4Returningto(2.6)wehavetosolveitandinsodoingapplythecorrelations(2.9)and(2.10)so
astoarriveat
maxljx(t)σχ2+(µs2+σξ2)|y|<ymaxdydτση2+(µs2+σξ2)dy(hiylix)◦hjy(−t)=µ2shjx(−t)
i<t<τ0(2.12)fordetailsseeAppendixA.3.Theopencircle◦denotestheautocorrelationintegral
∞(a◦b)(t):=dτa(τ)b(t+τ).(2.13)
−∞Inordertosimplify(2.12),wedefinetwonewnoisemeasures,
2τ2:=σ2ξ(2.14)
µsandσ2:=σχ2+dydτση2(µs2+σξ2).(2.15)
µs20|y<τ|<y<tmaxmaxµs2
Theparameterτrepresentsaninversesignal-to-noiseratio.Therefore,itisoftenreasonableto
assumeasmallvalueofτ.Theparameterσ,ontheotherhand,describestheoverallmeasurement
noisebyrelatingdetectionandtransmissionnoise,σχandση,tothesignalmeanamplitudeµs.A
priori,itsvaluecannotbeassumedtobesmallandhastobeadjustedaccordingtothesituationat
hand.Inordertofurthersimplify(2.12)weswitchtoFourierspace,whereconvolution(2.2)and
correlation(2.13)becomeordinarymultiplicationscombinedwithcomplexconjugations.Using
(2.14)and(2.15)anddenotingFouriertransformsbycapitallettersandthecomplexconjugation
byanoverline,(2.12)simplifiesto
Lixσ2δij+(1+τ2)dyHiyHjy=Hjx.(2.16)
i
17
OptimalMapFormation
Equation(2.16)isthemainresultofourderivation.Inprinciple,itallowsustocalculatethe
x–invseeerse(A.16)transferinthefunctionsAppLendixiffororoptimaldetails–signalthenconfirmsreconstruction.thatAthecalcinvuerselationofthetransformationsecondvweariationhave
foundindeedminimizestheerror.Forconveniencewewillintroduceanalternativenotationinthe
section.nextionnotatMatrix2.2.3Torewrite(2.16)inamorepracticalnotationweintroduce“matrices”HandLbyputting
H[ix]=Hix,L[xi]=Lix.(2.17)
Thenotationsillustratethattransferfunctionsandinversetransferfunctionsarelineartransforma-
tionsfromacontinuousspace(theoutsideworld)intoadiscretespace(theneuronalmap)and
viceversa.Therefore,HandLareonlyformallymatriceswithaspatialcoordinatexvaryingin
R.Thematrixmultiplicationinvolvingthespatialcoordinatemustconsequentlybeunderstood
asanintegration.Adiscretizationofspace,asisusualinnumerics,wouldleadtoatruematrix
ulation.formInaddition,weintroducethecovariancematrixC(R)ofthereceptorresponseRasdescribed,
e.g.,in[135,144].Inourcasewefind
C(R):=(R−R)(R−R)T=µs2σ21+τ2HHT(2.18)
wherethesuperscriptTdenotesthematrixtransposeand1theidentitymatrix.Equation(2.16)
nowsimplifiesto
MLT=HwithM:=µs−2C+HHT.(2.19)
GivenMasaninvertiblematrix,denotedasthe‘modelmatrix’,thesolutionforLturnsouttobe
sL=M−1HT=HTµ−2C+HHT−1.(2.20)
Thisequationgivesauniquesolutionfortheoptimalreconstructionforanygivensetoftransfer
functionsandnoiseconstants(σ,τ).Using(2.4)inmatrixformwefind
Sˆ=LR(2.21)
asestimatedsignalfromthemeasuredresponsevectorR.
2.3Relationtocommonmethods
Thechallengeofsignalreconstructionhasalongtradition,and,accordingly,onemayaskhow
theaboveformalismrelatestomethodsthathavebeenestablishedinthisfield.Inthefollowing,
wewilldiscusstherelationofourmodeltomethodsbasedonthepseudo-inverseandtothe
maximum-likelihoodapproach.
Pseudo-inverse.2.3.1Ifthenoisetermscanbeneglected,theinterpretationof(2.19)isstraightforward.Inthiscasethe
covariancematrixCvanishesandtheresultingequationleadsto
L=HT∙H∙HT−1.(2.22)
Thepseudo-inversereconstruction1ofH[12functions].AtLthathindsighwet,havethisjustmakesfoundsensefulfillsincetheptheropertiespseudo-inofvtheerseMoore-Pgeneratesenrosean
1TheexplicitexpressionforLin(2.22)onlyholdsif(H∙HT)isinvertible.
18
rmationoFMapOptimalinvapproersionximatemayinvnotersebepmatrixossiblethatforaminimizesmatrixHthethatquais,dratice.g.,error;rectangularseeAppinsteadendixofA.5.squareAnorexactof
Butincompleteevenrank.inthemoregeneralsituationofnon-vanishingnoiseterms,wecanobservestrong
therelationscalculationbetweofentheourframewpseudo-inorkverse,andamethodsregularizationbasedonhasthtoebeinpseudo-introvducederse.toThesuppresspointisnoisethat,terms,for
ttermypicallyα1tomakehigh-frequencyitmorevstableariations.[193,The268,269,so-called228].TheTikhonov-Milregularizedlerreequationgularizationthenaddsreadsapositive
α1+H∙HT∙LT=H.(2.23)
2Hence,Comparinginthisthispsecialequationcaseourwithgeneral(2.19)weapproacseehthatisαidencorrespticaltoondsmethoexactlydsusingtoourthetermTikhonoσifτv-Miller=0.
regularization.2.3.2Maximum-likelihoodapproach.
Thedata.Withinmaximum-likelihothemaximodum-lanalysisikeliho[o135d,sc144heme]isaonecommoncomputestooltheinthestiminulusthatterpretationistheofmostlikmeasuremenelyonet
givenasetofdetectorresponsesR.Experimentshaveshownthatoptimalornear-optimalstimulus
Acommethobinationsdofopticanmalindeedstimulusdescribecomseveralbinationsucphenomenahasofthesensorymaximproum-likcessingeliho[o67d,126approac,165h,is2,113,therefore202].
highlyrelevanttoneuronalinformationprocessingandoughttobeincludedintoourmodel;see
1.Chap.Themaximum-likelihoodapproachtriestofindthemostprobableinputsignalSgiventhe
(σdetector=∞).respWenoonseswR,assumeaknoawnlineartransferrelationfunctionH,andnoaprioriknowledgeaboutthesignal
sR=HS+χ(2.24)
withχrepresentingthenoise.WeassumethenoisetofollowaGaussiandistributionwithzero
meanTheandmethothedstandardminimizesthedeviationnoiseσχ.χ.Thatis,basedonthefundamentaldefinitionsofBayesian
statistics,itmaximizestheconditionalprobabilitydensityfunction
p(R|S)∝exp−2σ1χ2(R−HS)T(R−HS)(2.25)
withrespecttothesignalS.Thisleadstoalinearsystemofequations
S=HTH−1HTR.(2.26)
=:LML
Usingtheaboveassumptionsforourmodel,viz.,σs=∞,η=0,andξ=0,(2.19)reducesto
HHTLT=H.(2.27)
Totestwhetherthetwofiltersareequal,weinsertLMLinto(2.27).Applicationofthetransposition
rulesshowsthatwiththeassumptionsweusedL=LML,thetwostrategiesareidentical;for
lodetailscalizationwerefercapabilittoyofelsewherethe[135,lateral-line144,242sys,te236m,].forWewillinstance,useofthisfishminethothednext(2.25)ctohapter.quantifythe
19
OptimalMapFormation
Figureconnects2.2:tosevNeuronaleralmaprealizationneurons.ofTheunimomapdalneuronsmap(encoformation.dingtheEachlocationsensorx)(heremayhairreceivcellse(mlabeledultiple)i)
thiswconnectionsay,thefromeactransformationhsensor.lix(Eact)hcanbeconnectionreliablyhasawrepresenell-definedtedinastrengthneuronalandnetwtemporkoral[76].delayt.In
2.4Neuronalrealizationofthemodel
Inthissectionwerelatethegeneralmathematicalalgorithmofoptimalstimulusreconstructionto
ainSec.concrete2.2.2neuareronalfulfilledconintext.Wneuronalethereforeprochaessing.vetoThatverifyis,wfirsteneedwhethertochecthekwhethassumptionsertheweneuronalmade
quantitiesandfunctionsofoptimalmapformationareself-averaging.Tothisendwenoteonthe
onedescribhandedbthatyastofirincghasofticproneuronscess,isforcorrelatedinstance,withaGaussianneuronalone;inputweandwillthatseeinneuaronalminutenoisewhy.canOurbe
frameworkcancopewithanydistributionofneuronalxnoiseaslongasthemeaniszero.Onthe
theothermapshandassotheciatedoptimalwithinvdifferenersetmotransferdalitiesfunctionsandlihence(t)arereflectlearnepropdertiessynapticoftheconnectionsunderlyingbetlearnweening
protimecess.scalesforEffectivelearninglearningandissloindividualwbecauserealizationsitneedsofanmanyexternalindepsignalendentcanrepbesetitions.eparated.AccordInotheringly,
ofwtheords,prolearningcess;seeis[a145self-a].Asvmeneragingtionedprocessbefore,wherequanonlytitiesaverandagedfuquannctionstitieswithinenterthebyphtheysicalverymappingnature
proexploitcesstheareself-amathematicalveragingasframewwell;orkaspleasederivseeedalinso[Sec.35].2.2Inareconclusion,fulfilled.theconditionsneededto
xInsuchConsequenanarctly,whitecture,ecanthenowactualtranslateprothecessinginveisrseptransfererformedbyfunctionstheli(synaptict)intoconnectionsneuronalbhardwetweenare.
neuronsanddetectors.Spatialprocessingisgovernedbythetopographicstructureofthenetwork;
thatdeterminedis,whicbyhthedetectordistributionisconnectedofdelatysowhicwithinhtheneuron.setofTempconnections.oralproFigurecessing2.2onshothewsotheranhandexampleis
ofsuchaneuronalsetup.
andInthetheabensuingovederivmapationthroughweahavediscretealreadynumtakberenofintoinverseaccounttransferthefdiscreteunctions.charFacurthtereofrmore,detectorsthe
discrete,“spiky”characterofresponseandreconstructionbytheneuronalrealizationisalready
xthetakeninvcareerseofbtransferythefunnoisectionstermslx(tχ).iandThisλi.Thatdiscretizationis,weisarerealizedleftwithbyathesamplingtemporalprodiscrcedureetizationwhereofa
ixbnumeenbershoofwnthatdendritesawithlimitedapnumpropriateberofdelayssynapticischosenconnectionstorepresensufficestthetocompletesamplelithe(t).timeIthascourseindeedof
lix(t)[76].Evenmoreso,themap-neuronresponseisrobustwithrespecttothesamplingmethod
ofthetemporaldelays[178]aswell.
20
FMapOptimalrmationo
asimpleConsequenfeedforwtly,ardasnetwillustratedorkofinFig.excitatory2.2,ourandunifiedinhibitoryframeworkconnectionscanbineorderimplementoformtedbayunimomeansdalof
mappatternfromisarbitraryestablishedininpuat[real76,252,biological178].Itsystem.doesHerenot,thehowevcorrecter,explainsynaptichowconnecsuchtaionshavconnectivitetobye
learned.Ithasbeenshown[74,78]thatateachersuchasthevisualsystemcangeneratecorrect
synapticstrengthssothatamapcanindeeddevelopinothermodalitiesbymeansof(supervised)
STDP;compareforthedetailslearnedseeSec.connectivit1.2.3,ypSec.attern2.5.3,withandtheChap.optimal6.oneThanksastogiventhebypresen(2.16)tandmethod(2.20).wecan
Abiologicalmeansetup,ingfulthough,comparisonmayofnotthebestraighmathematicallytforward.Inoptimalrealnetwbiologiorkcalarcsystems,hitectureerrorwithanreductionactual
asin(2.5)toitsminimum–thatis,realizingtheoptimalconnectivity–maynotbepossible
becauseofneuronallimitations.Thelimitedneuronalaccuracythatresultscanbeincludedinto
ourframeworkbyreducingtheerroronlybelowacertainerrorthreshold,whichmayevenvaryin
space.Forinstance,thesamplingarraysofanimaleyesarenon-uniform,withdifferentpartsof
fothecusonvisualspfieldecificbeingspatio-tempsampledoralwithdomainsdifferentcanspatialmathematicallyandspectralbererealizedsolutionby[in125tro,257ducing,293a].pSucositivhea
wcertaineightingfuthreshold,nctiontheintoareathesinwithintegraltheoffo(2.5).cusoftheAccordinweighglyt,whenfunctionhareducingvetothereachglobalahighererrorbleveloelwofa
optimization,i.e.,ofresolution,thantherest.
AsindicatedinSec.1.2,theconceptofreceptivefieldsisincludedinourformalism.Itmaybe
goodtoremember,though,thattherearetwomappingfunctions(LH)andLprojectingdirectly
ontothemap.Sincetherowsofthesemappingfunctionscontaintheinformationfromwhichareas
aofspreceptivecificemapfieldsneuronwhenwreceiveescomeintoput,theauditoryrowsdessignalcribproethecessingreceptivineChap.fields.3.WTheere,willweusetaktheeaconceptcloser
lookattheneuronalimplementationbymeansofaconcretesettingandexploittheconnection
betweenoptimalmapformationandreceptivefields.
Takentogether,theformalismofoptimalmapformationiscapabletodeliveranoptimal
neuronalconnectivitypattern,justasillustratedinFig.2.2,andherebydirectlygivesaforecastof
howexampletheinthereceptivconetextfieldsofaretheshapvisualed.Insystemthetonextillustratesection,theweprocapabilitiesvideavoferythesimplepresentbutframewconcreteork.
Spatialexample:visualprocessing
Withinthevisualsystemeachsensoryneuronisbasicallytunedtoaparticularspatialposition.In
mathematicalterms,everyretinalneuronireceivesinputfromaspatialpositionxi,itspreferred
position,andneighboringpositionswithinaregiondeterminedbyresolutionρ.Thetransfer
functioncorrespondingtosuchasensorysystemis
2hix(t)=exp−|x−xi|δ(t),(2.28)
2ρ2
anditsFouriertransformreads
(2.28)
2Hix=exp−|x−x2i|.(2.29)
ρ2Withinourexemplarysetupweassumethatthesignalpositionx=(u,v)encodespositions
u,v∈[−1/2,1/2].Asaremind2er,wehaverescaledpositionssoastomakethemdimensionless
andfitinthesquare[−1/2,1/2].Fromtheaboveansatz(2.28)and(2.16)wecalculatethematrix
21
OptimalMapFormation
Figure2.3:Spatialreceptivefield.Connectionstrengthstoamapneuronencodingtheposition
(u,v)=(0.1,−0.2).Thesensoryneuronsaredistributedona40×40gridwithpreferredpositions
u,v∈[−1/2,1/2]andatuningcurvewidthρ=0.9.Wechoseσ=1andτ=0.Aclearcenter-
surroundreceptivefieldemerges.Receptorneuronsthathaveapreferredpositionmatchingthat
ofthemapneuronhaveexcitatoryconnections(whitespot).Receptorneuronshavingaslightly
off-setpositioninhibitthemapneuron(darkcircle).Neuronswithpreferredpositionsfaraway
fromthemapneuronhaveconnectionstrengthzero(gray).
tsonencomp2
Mij=σ2δij+(1+τ2)exp−|xi4−ρ2xj|×erfui+2uρj−1−erfui+2uρj+1
(2.30)vi+vj−1vi+vj+1
×erf2ρ−erf2ρ
whereerf(x):=√2π0xexp−y2dyistheerrorfunction.Tofindtheconnectionstrengthslix,
wenumericallycalculatethemodelmatrixMforadiscretizedspaceandparametersσ=1and
τ=0.transformationWiththewematrixcannMumerwiceallythenobtaindeterminelxfortheeachmapconnectionpositionstrengthxassshoL.wnByinanFig.inv2.3.erseFHereourithere
iconnectionsfromallreceptorstoamapneuroni,i.e.,itsreceptivefield,areplottedforanarbitrary
strongpreferredpropjectionsositiontoxithe=(0.map1,−0.neuron2).(brighClearlyt,sptheotinFig.receptors2.3)encobut,dingintheterestinglypreferred,thepreceptorsositionhathatve
encodeslightlydifferinglocationscontributenegatively(darkcircleinFig.2.3).
Suchacenter-surroundprofileiscalled“Mexicanhat”andis,e.g.,realizedbylateralinhibition,
awell-knownphenomenonfirstdescribedbyMach[184]inthevisualsystemin1866.Uptonow
forthismecinstance,hanism,insescttudiedvisionin[the130],snakmammalianeinfraredvisualvisionsystem[255[,281252,],143],electrichasbeenfielddiscodetectionveredinaswellelectricin,
fish[250],andsurfacewavedetectioninthebackswimmer[203].
surroundIncontrastreceptivtoemanfieldyismoadelsnaturalsuchasconsequencepop-outof[153our]moordelsaliencyandthusdetectionexplains[129,lateral175]ainhibitioncenter-
asandoptimalapplyforourconceptmap-formationofoptimalpurpmaposes.Informationthefollotowingelabcoratelyhapter,wderiveevisanitarcthehetypauditoryemodelsystemfor
signalenhancementandechosuppression.IntheAppendixA.1,weadditionallyprovideaneasy
step-by-step“recipe”soastofindtheoptimalconnectivityinanarbitrarybiologicalsetup.
Next,weleavetheframeworkofoptimalmapformationforthetimebeinguntilweconsider
thethesensoryinfraredpsystemerformanceofsnakofesthe(Chap.auditory5)sandystemrevisit(Chap.general3),asptheectsoflateral-linecross-mapsystem(Chapcomputation.4),thatand
followfromcombiningmono-sensorymaps.
22
MapOptimalrmationoF
interactiondalMultimo2.5Inthelastsection,wegainedadeeperunderstandingofmonosensorymapsandtheircomputational
power.Thatis,wecanrevisitatopicofSec.1.2,Cross-mapcomputation,andtakeacloserlookat
thisimportantissue.Onthelevelofmaps,wegenerallydistinguishtwocategoriesofmultimodal
interaction,viz.,cross-mapcomputation:integrationandpoolingofinformation.
Integration2.5.1Congruentspatialinformationfromdifferentsensorysystemscanbeintegratedintoasinglemerged
and,hence,multimodalmap.Suchanintegratedmap,ascomparedtounimodalinformation
processing,featuresincreasedinformationreliabilityandsaliencyaswellasanimprovedsensitivity
inbothspaceandtime[166,167,104,237].Forexample,ifvisualandauditorysensorysystems
bothregisterasignal,e.g.,“brownahead”and“barkingahead”,itisveryprobablethatthesignal
correspondstoanactualobjectratherthantoasensoryartefact.Atthesametime,theintegrated
signalwillbestrongerandallowsforfasterreactions(e.g.,“escape!”).Insomecasesanintegrated
signalisevenoptimal[102,202].
Moregeneralneuronalmodelsdescribingmultimodalintegrationandbasedonstatistical
methodshavebeenpresentedelsewhere[59,60,61].Concretetheoreticalmodelsofmultimodal
integrationwithintheSChavebeendevelopedaswell[7,220,238,239,186,274].
olingoP2.5.2Notonlycanthemonosensorymapsbemergedintoamorereliablemultisensorymap,butthe
diverseinformation,thus,signalcharacteristicswithinthemonosensorymapscanbeaccessed
simultaneouslyaswell;rememberFig.1.5.Thissimultaneousaccessingisonlypossiblesince
allmonosensorymapsarealignedandconsequentlyspace-timecanservetolinkthedifferent
modalities.Consequently,anobjectatonespecificpositioncanbeidentifiedandcharacterizedin
ordertoselectmotorresponsesinacomplexenvironment.Forexample,arattlesnakemaydetect
spatialcoherentactivityinitsvisualand/orinfraredmap.Onlyiftheencodedobjectisvisible
andwarmitwillbeidentifiedasalivingpreyobject.Ifitisvisibleandnotwarmthesnakewill
discardneuronaltheANDandinformation.ORproExpcessingerimenstepstalforevidencethecomforsucbinationhapofoolingvisualofandinformationinfraredismappro[210vided,211by].
TheseprominentexamplesofpoolingintheSC/OTcouldenabletargetselectionandthusensure
appropriatemotorcommandsinacomplexenvironment.
Despiteincreasedreliabilityofanintegratedmap,itsindividualinputstreamscannotbe
distinguishedanymore.Thatis,theinformationaboutwhichmonosensorymaphasdetermined
thepositionislost.Withintheaboveexamplethemultimodalmapmayindicateamultimodal
eventahead,butthetriggeringmodality,thatis,visual,auditory,oryetanothermodalityremains
ed.unresolvInsummary,integrationofinformationallowsforareliablespatialdeterminationofanobject,
thekeytaskofobjectformation.Ontheotherhand,poolingofinformationassuresanaccesstothe
detailscorrespofondsantoobjaectswitchnecessarybetweenforobjeparallelctandidentificserialation.dataSwitcprohingcessingbettowbeenestinfittegrationdifferentandtasks.pooling
Toenableefficientmultimodalinteractionsuchasintegrationandpooling,alignmentofthe
differentmono-andmultisensorymapsisofcrucialimportance.Onlythencanamultimodal
stimulusataspecificspatiallocationbeidentified.Analignmentofsensorymaps,however,isnot
presentatbirthandmustbelearned[116,262,156],asdiscussedinthenextsection.
23
OptimalMapFormation
Development2.5.3Wenextcomebacktothequestionofhowsensorymapscanbealigned,i.e.,howacommon
multimodalspacecanevolvethatwehavebrieflyintroducedinSec.1.2andSec.1.3.Anobvious
solutiontosuchanalignmentprocesswouldbetheexistenceofonedominantmodalityasreference
forallothermodalities[158,156].Thisreferencemapwouldthenautomaticallyleadtomodifications
maps.otherallofAndindeed,experimentalandphysiologicalstudieshaveshownthat,inmanyanimals,destruc-
tionordisturbanceofthevisualpathwayleadstodisorganizedandabnormalsensorymapsin
non-visualmodalities.Thesefindingshavebeenobtainedinhamster[200],cat[280,278],clawed
frog[44],ferret[150],barnowl[159],andinsnakes[96].Psychophysicalexperimentswithcongeni-
tallyblindandnormallysightedhumanshaveshownthatvisualinputearlyinlifeisnecessaryfor
multimodalinteractiontooccur[122,230,233].Consequently,visionseemstoserveasa“teacher”
dalities.monon-visualforAplausibleargumentsupportingtheideaofvisionasteacherinputistheintrinsictopographic
orderoftheretina.Itisknownthatlayersofneuronscanself-organizeintotopographicmaps,
proreadervidedisthatreferredtoinitially[271a].smThisallsettogetherofcorrecwithtlytheorganizedsubsequentneuronsdevelopmenexists[t284of].layFeorrsainthereviewvisualthe
cortex(formice,see[133])mayallowtheintrinsictopographyoftheretinatostep-by-stepdictate
theorganizationandalignmentofhighervisualand,potentially,alsomultimodalmaps.
Thegeneralmechanismfacilitatingtheprecisionofsuchanalignmentofmapsisspike-timing-
dependentplasticity(STDP)[16,17,55,87,115,145,188,254,294].Anexamplewherethe
alignmenthasbeenstudiedindetail,bothexperimentallyandtheoretically,isaudio-visualintegra-
tionwithintheOTofthebarnowl.Hereexperiments[127,158]haveshownthattheauditorymap
followssystematicchangeswithinthevisualinput.Althoughtheprecisenatureofthisteaching
signalhasnotbeenclarifiedexperimentally,selectiveneuronaldisinhibition,orgating,seemsto
playakeyrole[103,285].Althoughexcitatoryandinhibitoryteachinginputcantheoretically
accountforpropermapalignment[78,56]onlyinhibitoryteachinginputisabletore-alignalready
[78].connectionsmapestablishedInsummary,theabovestudiessupporttheideaofvisionasteachermodalitytoalignother
monosensorymaps,buttherearecontradictingfindingsaswell.Wecansummarizethesefindings
intotwomajorpoints.First,visionisnotneededatallasteacherinputforthelearningprocess
ofsensorymaps.Second,visionshowsplasticityasitisinfluencedbyothermodalitiesandasit
improvesduringdevelopment.Concerningthefirstpoint,ithasbeenshownboththeoreticallyand
experimentallythat,althoughimprecise,amapofazimuthalsoundlocationcanbelearnedwithout
anyvisualinput[146,158]thoughadmittedlyonageneticallydeterminedsubstrate.Inaddition,
non-visualmodalitiescaninfluenceeachotheraswell,e.g.,auditioncaninfluencehaptics[31].
Moreover,somatosensoryreceptivefieldsalreadyshrinkinapostnatalphasewhenonlyauditory,
butnovisualneuronsarepresent[279,277].Forthesecondaspect,behavioralandpsychophysical
studiesshowthatvisualperceptioncanevenbeinfluencedbyothermodalitiessuchashaptics[68]
oraudition[249,77,258].Moreimportantly,visionitselfcanimprove,respectivelysharpen,as
foundinthevisualsystemofyoungcats[279,277].
Altogethertheexperimentalandtheoreticalfindingswehavepresentedabovequestionthe
currentpictureofvision-guidedmapalignment[148].WallaceandStein[279]havepointedout
thatthedevelopmentofdifferentmodalitiesstartsinparallelandintemporalcoincidencewiththe
appearanceofmultimodalintegration.Theyherebysuggestacommonmechanismdrivingboth
mapdevelopmentandmultimodalintegration.InChap.6,wewillsticktothisidea,namelythat
acombinationofallavailablesensoryinputintrinsicallyguidesitsalignmentbydevelopingthe
conceptofintegratedmultimodalteachingthataddressestheissuesmentionedabove.
Inthenextchapters,weturntomoresophisticatedsettingsthaninSec.2.4andgainnew
insightsintothecapabilities,characteristics,andlimitationsofthesessenses.
24
Esh¨ortdochjedernur,waserversteht.
JohannWolfgangvonGoethe
3.Auditoryprocessing:Exemplifyingoptimality
Inthischapterweapplyourmodelofoptimalmapformationtotheauditorysystem.That
is,signalweprodeducecessing,anaviz.,rchetypehandlingmodelechoandes.coBasedrresponondinourgmotempdel,oralwepreceptiveredictthefieldsabilitforyofauditobothry
mathematicalandneuronalimplementationstosuppressechoesandextractsoriginalsignals
foinrm.vaInriousaddition,echo-richwecscenaomparios,reevenstate-of-the-ainthertabsencetechnicalofexactalgoinforithms,rmationbiologicalonthesolutionsspecificandechoour
atherchetcompaypemorisondelhintsderivedtofromlimitationstheofgeneralourgeneralframewoframewrk.orkOriginandatingpromisingfromthissitesparfoticrulafuturerwsetting,ork.
ductionIntro3.1inIntheChap.pres2toentachapterconcretewesettingapplyinthetheframewtempororkalofdomain.optimalInmapordertoformationdoso,wewetakepreviouslytheauderivditoryed
pathwayasareferenceforanarchetypemodelofsignalenhancementandechosuppressionthat
yieldoptimaltemporalreceptivefields.Togetstarted,weinitiallyreviewtheessentialfactsand
specificquestionsofinterestaccordingtothissetting.
Innaturalenvironments,auditorysignalsarefollowedbyechoes,orreflections,thatdegradethe
signal.Inthenon-biologicalfield,thissettingiscalledreverberation.Notsurprisingly,biological
systemsprocessingacousticsignalshavedevelopedneuronalmechanismsofechosuppression
[20after,221the].originalHumans,signal,fordepexample,endingdoonnottheconsciouslysignalcpharactererceive[ec20ho].esWearrivingthereforelessexpthanectabdelaoutys20andms
suppression,inneuronalterms,inhibition,toshapethereceptivefieldsofourmodel.Fora
ofquanthetitativreceptiveundeerfieldstanourding,mohodelwever,predicts?moreHospwdecificoesthisquestionsshapeneedconnecttobetoasked.biologicalWhatisrealitthey?shapCane
differentfunctionsbeattributedtodifferentregionsofthereceptivefield?Canweidentifyuniversal
characteristicssharedbydifferentreceptivefields?
Wehavealsochosenthecontextofauditorysignalprocessingbecausethisisasitewhere
hatecvehnicalalreadysolubeentions,realized.viz.,Thisalgorithmspromisesthataaddressfruitfulsomecomparisonoftheofabothevemoquedelsstionsandtoabiology.certainAlthoughextent
filters,artificialsimplealgorithmsandarerobustquitealgorithmssophisticatedmaywoutperemarkerformthatmore“[...]forsophisticatedacousticecsolutionsho[...]”cancellation[105].
So,thereareartificialalgorithmswithgoodperformancethatprovideenoughsimplicitytobe
settingcomparableandtoconthetrastarcourhetypmoedelmodelandatitshand.collectionIndoingofso,receptivweeapplyfieldsourtowhatpresenwtemofinddelintoanatureneuronaland
technics.Finally,similaritiesanddifferencesofthederivedarchetypemodelmaypointoutinwhich
wayalgorithmscanbeimprovedbymeansofbio-analogouscognitivearchitectures.
25
SignalEnhancementandEchoSuppression
3.2Archetypemodel
Inthissectionwebrieflyrecapitulatetheessentialstepsofourframeworkofoptimalmapformation
intheconcretesettingofauditorysignalprocessing.Thatis,wederiveanarchetypemodelof
signalenhancementandechosuppressionbasedonlyonthemathematicalprincipleofoptimality.
3.2.1Optimality:Anarchitecturalprinciple
Themodelderivationisbasedontheideaofsignalreconstructionbyoptimalreceptivefieldsthat
inversethecorruptionprocessofechoesandnoise.Byderivingthenecessaryneuronalconnection
strengthsweformthebasisforasmoothintegrationofourarchetypemodelintotheexisting
knowledgeonauditoryprocessingsuchasrelatedphysiologicalandpsychophysicalresults.
Forthestartingpoint,themodelminimizestheexpectationvalueofthequadraticerror
E:=(st−sˆt)(st−sˆt)betweentheoriginalauditorysignal,s=(...,sµ,...),anditsoptimal
reconstructionsˆ=(...,sˆµ,...).ThereconstructedsignalsˆminimizesEandisthusoptimalin
theleast-squaresense.Thisisthedefinitionofoptimalitywehavealreadyintroducedpreviously.
Theoptimalreconstructionsˆiscomputedbythetensorl.Itscoefficientssufficethesetof
equations(∂E/∂lµν=0).Theresultingreconstructionalgorithmcomputessˆonlybymeansofthe
sensoryresponser=(...,rµ,...)oftheauditorysystem,sˆt=ltτrτ.Thecoefficientsltτofthe
matrixlaredescribinghow(inneuronalterms:withwhichconnectionstrength)avalueofthe
incomingsignalatthetimet−τismappedontotheoutputsignalattimet.Neuronallythiscan
berealizedthroughadelaylinewithdelaytimeτandasynapseofsuitablestrength.So,lcontains
thesetofoptimaltemporalreceptivefieldsthatcompensateforthecorruptionoftheoriginalsignal
onse.respsensorythewithinThesensoryresponseitselfwillbe,inprinciple,aconvolutionoftheoriginalsignalandthe
transferfunctionhthatinthissettingwedenoteasechofunction,viz.,rt=htτsτ.Inthetechnical
fieldtheechofunctionhismorecommonlyspecifiedas“roomimpulseresponse”(RIR).Itdepends
onthephysicalsurrounding,e.g.,the“room”.Inanalogy,thescenariobiologicallydenotedasecho
suppressionistechnicallyreferredtoasde-reverberation.
Asinthegeneralcase(seeChap.2),anyauditorysensorysystemalsohastocopewith
uncertaintiesnomatterwhethertheycomefrommeasurementerrors,varianceswithintheassumed
physicaltransmissionprocess,orsimplythroughthefactthatspaceandtimearecontinuous
quantitiesthatsensorsandespeciallyneuronscannotcontinuouslyrepresent.Artificialauditory
sensors,forinstance,typicallyaverageoveraspecificamountoftimeandgiverisetomeasurement
errors.Thetemporaldynamicsofneuronsareeveninherentlylimitedbycharacteristicssuchas
refractoriness.Inconsequence,theinput-outputrelationwithinanysystemwillbecorruptedby
errorsornoise.Wethusrewritethesimpleconvolutionrt=htτsτbyaddingthenoisetermς
accountingformeasurementandtransmissionfailurestooursensoryresponse.Inaddition,our
signalofinterestmay,andingeneral,willbedisturbedbycomplementarysignalswepoolandrefer
toasbackgroundnoisewemodelbyaddingarandomvariableαtothesignal.Wehencearriveat
rt=htτ(sτ+ατ)+ςt.(3.1)
Beforecalculatingthereceptivefieldslweneedtodiscussiftheassumptionsregardingthe
expectationvalueswehavemadeinderivingthegeneralframework(2.16)arevalidinthepresent
contextofauditorysignalprocessing.Ofcourse,theinputsignalstisdeterministicandwewould
notexpectanyproblem.However,wedonotknowexantetheexactvaluesofst.Wecanovercome
thisproblembylookingat“biologicallyrelevant”signals.Suchsignalsbelongtoaclassofsignals
thatwedenoteas“typical”.Consequentlyaspecificsensorysignalisaconcreterealizationofa
classoftypicalsignals.Thatis,theinputsignalstisastochasticquantitywithadefinedmean
µSandweuseanexpectationvaluetocomputeEinthesamewayasinthegeneralframework.
26
SignalEnhancementandEchoSuppression
Furthermore,wealsotakeallappearingcross-correlationstobezero,thatisnowthesignalvalue
atonespecificpointintimedoesnottellusanythingaboutthesignalvalueinthenexttimeframe.
Thesameholdstrueforbothtypesofnoise.Wetakethenoisetobeindependentatdifferent
pointsintime,eachwithstandarddeviationσς/α.Pleasenotethattheallocativefunctionforthe
noiseneednottobeGaussian.We,asexpected,canthereforeusethesameassumptionsasincase
of(2.16).Thatis,onlytheautocorrelationsremainandwithδµνasKroneckerdeltaaregivenby
sµsν=µs2δµν,ςµςν=σς2δµν,αµαν=σα2δµν.(3.2)
Wefinallyobtainalinearequationforthecoefficientsofthereceptivefieldsldependingonly
onthetransferfunctionhandthedimensionlessparametersσ:=σς/µSandη:=σα/µS,
lµγhνδhγδ1+η2+σ2δνγ=hνµ(3.3)
wherehνδisthetransposeofthematrixhγδ.Theparametersσandηcorrespondtoinverse
signal-to-noiseratiosofσ–themeansignalstrengthtothevarianceofthedetectormessurement
errors–andofη–themeansignalstrengthtothevarianceoftheacousticnoise.Thereceptive
fieldslnowmatchtheechofunctionhandallowcalculatingthereconstructionsˆoftheoriginal
auditorysignals.Sincewehaveneitherspecifiedtheoriginalsignalnorusedanyinformation
suchasinputcorrelations(3.2),ouralgorithmcanreconstructanyarbitrarysignal.Togetan
optimalreconstructionperformanceinthecaseofnoise,wesimplyneedtoadjustthetwomodel
parametersσandη.Ifwehaveaccesstothenoiselevelsσς/αandthetypicalvalueoftheoriginal
inputstrengthµS,thedefinitionofσandηgivesusagoodestimateofthesevalues.Incaseprior
knowledgeofintrinsiccorrelationsoftheoriginalsignal,wecanincorporatethisintoµSthathence
transformsintoamatrixwithoff-diadonalelementsandleavetheaboveminimalansatz.
Wehavenowderivedanarchetypemodelfromourframeworkofoptimalmapformationina
concretetemporalsetting.Inthenextsubsequentsectionwewillanalyzetheperformanceandthe
robustnessofthemodelbyapplyingittoexemplaryechofunctions.
3.2.2Receptivefields:Acomputationalbuildingblock
Byanalyzingthemathematicalmodelderivedintheprevioussectionwecanobservetheintrinsic
capabilitiesandcharacteristicsofourapproach.Wedealwiththreeexemplarybutexplicittemporal
transferfunctions,namely,theechofunctionshd;r;e;cf.Fig.3.1.Indoingso,wetakeadelta
functioninthetimedomainasinputsignal.Thisfunctioncorrespondstoaclick.Itappears
respslightlyonse.Thesmearedsimplestoutandenvirisfolloonmewntedbyfeaturinganecanhoecashowdefinedouldbbyeathesingleechosolidfunctionwallininathefreesensoryspace.
ecHere,hothefunctionechowewouldassumebeaasingle,simple,wdiscreteeakenedrreflectionepetitionofaofclicktheandoriginlabelalthesignal.echo“d”Henceforforthe“discrete”.first
Thesecondechofunction,“r”for“realistic”,mimicsatypicalroomimpulseresponse(RIR)where
wehavenotonebutmanywallsandwherethesignalisfollowedbyashortsilenceandmany
thesubsequenclick.Ftorthereflectionsthird[ec170ho].Wefunctiontakewetheseassumereflectionstheclosedtohavespaceatomaximbeevumenatmoreabout20restrictedmsafterand
hencedonotsupposeanygapbetweensignalandreflections.Herewetakeanexponentialdecayof
thereflectionsandthereforemarkitwith“e”.So,roughlyfollowingsomeexemplaryRIRs[105],
wecovertherangeofpossiblecomplexechofunctionsbythreesimplifiedscenarios,i.e.,explicitly
assumingthereflectingboundariesatτveryfar(case“d”),normal(case“r”),andclose(case“e”)
distance.Intensornotation(rt=htsτ),theyare
hdµν=exp−(µ−2ν)+exp−(µ−ν+2βd),(3.4)
22
2κ2κ
hrµν=exp−(µ−ν)2/2κ2+A(ν,µ),(3.5)
27
SignalEnhancementandEchoSuppression
Figure3.1:Threevariationsofechofunctionshwiththeoriginalsignalsandtheresultingsensory
respodiscretizednser;echosettingsfuncti“d”,ons“r”,hd,r,eand(bl“e”ack=from0,leftwhiteto=righ1t.)inThematrixuppernotationgraphs(asA−Cdefined)depictby(3.4)the
–(3.6).Thelowergraphs(dD,r,−eF)showtheoriginalsignals(adeltafunction,filledline)and
resultingexemplaryecsensoryhorespfunctions,onsesvriz.,(soliddiscreteline)(left),inrealisticarbitraryunits.(middle),andThroughoutexponenthistialpaper(righwet)useecho,threeas
.2.3.2Sectioninexplained22heµν=expexp[−−((µµ−−νν))/κ/2]κififµµ−−νν<>00(3.6)
2wheredefinesalltheconstancommontsarerealsynaptic(∈Ralph).aThefunction,functionµAand(ν,νµ):=denote(µ−theν+roβrws)expand−(µcolumns−ν+ofβr)h/,2γand
(µdela−yνb)etweindicatesensignaltheandrelativechoefordiscretizedconditiontime“d”anddifference.“r”.TheTheconstanparameterstsκβanddγandβdenoterdenotehowthethe
signalsandechos,respectively,getbroadened,andκisameasureforthedecayoftheexponential
“e”.onconditiintailWenowwanttoarriveatathoroughunderstandingofthereceptivefieldsinvolvedinour
theoptimalthreemoechodel.Tofunctionsthisend,definedwebhayve(3.4)–calculated(3.6);cf.sensoryFig.3.1respandonsesFig.and3.2.Wreceptivehaevefieldsalsoforeveacaluatedhof
theincludingrobustnessnoiseofandtheamo“mismdelbatcyh”comparingconditionwherereconstructedechofuandnctionoriginalandsignreceptivaleforvfieldsariousdonotconditionmatch;s,
cf.Figs.3.2and3.3.Importantly,wehavetestednotonlythecompleteoptimalreceptivefieldsas
derivedintheprevioussectionbutalsoanabridgedversionthatdoesnotrequireanintegration
withwindoandwandwithoutthereforenoise,allowwsorkpropreal-timeerly;profordetailscessing.seeBothFig.3.2.abridgedandcompletereceptivefields,
andl.MoreoInver,doingwesohawveeusetestedthethewrongcollectionreceptivofefieldsreceptivforefieldsinreconstructingcaseoftheasignalmismatchandbetwcompareeenh
Fig.3.3reconstructionandshopwthaterformancethewithreceptivthee“matcfieldshing”(mainlylrcondition.)areAllableninetopcopeossiblewithcasesextremearevdepictedariationsin
–namelythewrongecho.Thesignalcanbereconstructedbymeansofreducingthenoiseand
effectivelysuppressingtheecho;cf.Fig.3.3.Thisflexibilityisaprerequisiteforanybiologicalsystem
sincescenario.suchItastysurnstemoutcannotthataffordincreasingaspσecificmakesneuronalthewirindifferengtforevreceptiverypeossiblefieldseclesshosensible,(suppression-)more
robust,andthusmakesthemperformbetterinthe“mismatch”condition,cf.Fig.3.3.
28
SignalEnhancementandEchoSuppression
toFigureright.3.2:GraphsReceptiv(Ae−C)fieldslrepresenandtthereconstructedcompletesignalsreceptivsˆ;eforfieldscondild,r,etionas“d”,defined“r”,inand(3.3)“e”(grafromy)andleft
theabridgedversions(black).Theversionsd,r,andealwaysappearasleft,middle,andright.The
areshoreconstructedwningraphssignals(Dobtaine−F)d.Thethroughbcompleteothabridgedreceptive(blacfieldsk)landleadtounabridgedhigherp(graeaky)receptivamplitudesefieldsand
lessdepictedartifactsin(Gin−theI)withreconstructed10%noiseassignals.comAparednoisytosensorinputstillstrength.allowsAllforagraphsreliablehavebeenreconstructionexpressedas
inarbitraryunits.OriginalsignalandsensoryresponseareasinFig.3.1.
Toanalyzewhichpartofthereceptivefieldscorrespondstowhicheffectinsignalreconstruction,
wehavesetselectedentriesinthereceptivefieldstozeroandobservetheconsequences.Inresult,
e.g.forthe“realistic”receptivefields,thefirst,fastinhibitoryregionsharpensthecontourofthe
signalpeakandreducestheexponentiallydecayingtailoftheecho.Thesecond,slowinhibitory
regionreducesthesteepnessoftheechoonsetthatisimportantforauditoryobjectformation.
Tosummarizethepresentsection,wehavederivedandanalyzedasimple,yetgenericarchetypeof
modelforauditorystimulusreconstruction.Inthenextsectionwebrieflyreviewthestate-of-the-art
approachesintechnicalauditoryprocessingandrelatethemtothemodelathand.
3.3Technicalprospect
Havingderivedanddiscussedaveryfundamentalmodel,wenowrelateittomorerecentapproaches.
Fromthetechnicalpointofview,whatwehavederivedisasinglechannelde-reverberationalgorithm
thatusesonlyonefrequencychannel.De-reverberationiswellstudiedandanotyetcompletely
solvedproblemintechnicalaudioprocessing.
Ourapproachcoversawiderangeoftypicalde-reverberationalgorithms.Forexample,htτ=δtτ
correspondstoaconventionaldenoisingscenarioasdescribedby[66,290].Foraspecificecho
functionandσ=η=0themodelinverts,or,foraσ=0approximatelyinvertstheequivalentRIR
intheleast-squaresenseasdiscussedbyMiyoshietal.[195].Aspecialcaseoflinearfilteringisthe
parametersetofη=0andσ=1thatcorrespondstoasimpleWienerfilter[135,228,232,236];
forareviewonlinearfilteringsee[141].
29
SignalEnhancementandEchoSuppression
actualFigureec3.3:hoFlexibilitfunctionyhof(gratheymoarea)del.inAllarbitrarygraphsshounits.wtheIneachreconstructedcolumnasignaldifferensˆ(blacteckholine)functionandtheh
isusedtocalculatethesensoryresponsewhereasthereceptivefieldslarevariedineachrow.
Echomismatcfunctionhingtheandactualreceptivechoefieldsfunctionmat(Bc,hC,onD,Fthe,G,Hdiagonal)can(A,leadE,toI).Evresonableenusingresultsifrecepσtivisechosenfields
appropriately.Hereσ=5.Inthecasemismatchincreasedσleadstoresultsmuchbetterthanthe
initialσ=1whichcorrespondstotheWienerfilter[135,228,232,236].
Forabriefsynopsis,wehavetriedtosubdividethevarioustechnicalalgorithmsforde-
reverberationinthreegenerations.Thefirstgenerationconsistsofstraightforwardmethods,
thesecondcombinesseveralmethodsfromthefirstgeneration,andthethirdgenerationtriesto
exploitthesamesignalcharacteristicsthatareexploitedbybiologicalsystems.Singlechannel
(usingonlyonefrequencychannel)de-reverberationofthefirstgenerationcanbedistinguishedinto
threemajorgroupsofalgorithmsbasedonde-convolution,blindde-convolution,andsuppression.
Thefirstgroupassumesknownechofunctions(i.e.RIR)anddealswithapproximateandfast
methodsforinvertingit[105,195,206,215],asrealechofunctionsareusuallynotexactlyinvertible
[206].Thesecondgrouptriestofindafilterwhichaccountsforacertaincriterion,forinstancethe
meanpowerspectraldensityofspeechornoisereduction,andindirectlyreducesthereverberation
[66,73,90,105,273,290,291].Asinthepreviouscasethisfilterconvergestoanapproximation
oftheechofunctioninversion.Thethirdgroupestimatestheamountofechopowerspectral
density[22,105,173]andusessuppressionmethodstoperform,forinstance,minimummeansquare
estimationoftheoriginalsignal.Inthestateofartalgorithmsforechosuppression,theoneswerefer
toassecondgenerationechosuppression,differentmethodsarecombined[79,105,110,288].An
exampleforthethirdgenerationareattemptstorelyechosuppressioninanunknownenvironment
onthepitch[272,273].
Butstillthesealgorithmsdonotmeettherequirementsforartificialcognitiveauditoryprocessing
suchassoundsourcelocalizationorspeechrecognitioninadynamicreverberatingenvironment.In
contrast,oureverydayexperienceprovesneuronalalgorithmsthatallowsophisticatedauditory
processingexist.Therefore,inthenextsection,weshowhowthepresentedarchetypeforecho
suppressioncanbetransferredtoaneuronalsystem.Bycontrastingtheneuronalrealizationwith
actualneurobiologyweidentifysimilaritiesanddifferences.Thisallowstopinpointpromisingsites
ofcognitiveprocessingintheauditorypathway,thatis,thesitesforfutureresearch.
30
SignalEnhancementandEchoSuppression
Figure3.4:Influenceofparameterσonthereceptivefieldslfor“d”,“r”,and“e”formlefttoright.
Thereceptivefieldsld,r,eforσ=1(black)andσ=5(gray)inarbitraryunits.Asσincreases,the
receptivefieldsgetsmearedoutandlosefinestructure.Simultaneouslythereconstructiongains
generality,whichisadvantageousforreconstructingasignaldeformedbyanunknownechofunction,
seeFig.3.3.OriginalsignalandsensoryresponseareasinFig.3.1.
ectrosppBiological3.4Inthecourseofthisthesiswehavestatedthattheframeworkofoptimalmapformationcan
beimplementedtoaneuronalsettinginastraightforwardway.Inthefollowingweprovethis
statement.Ourpresentmodelofauditoryprocessingallowsaneasyone-to-onematchofthe
optimalreceptivefieldswehavederivedabovetotheneuronalconnectionsincludingtheexplicit
valuesofdelayandsynapticstrength.Indoingso,wemapthereceptivefields(seeFig.3.4)toa
feedforwardnetworkasdepictedinFig.3.5.Suchaneuronalnetworkisthencapableofextracting
theoriginalsignalandsuppressingechoes.Theperformanceoftheneuronalrealizationisdepicted
inFig.3.6fordifferentsituations,viz.,receptivefields,andvaryingσ.Justasexpected,thesignal
isextractedreliably.Thatis,weobtainaneuronalmodelthatsuppressestheechoandextractsa
signaljustasinthemathematicalmodel.Weherebydonotstatethatechosuppressionisdone
inbiologybymeansofasinglefeedforwardnetworkbutgiveanexistenceproofofourmodelin
are.hardwneuronalAcomparativelookontheshapeoftheoptimalreceptivefieldsrevealsthattheresulting
synapticstrengthsofthedelaylinesfollowveryplausiblepatterns.Themostobviousfeatureisthe
emergenceoftwodistincttimescalesofsuppression.First,adirect,fastinhibitionwithaclear
maximum(e.g.atabout2mslastingca.5ms;cf.Fig.3.4B).Second,adelayedshallowinhibitory
region(inFig.3.4Bfromabout10-17ms)thatisthereforecalled“slowinhibition”.Thesetwo
findingsfitverywellintotoday’spictureofauditoryprocessing.Gapdetection,auditorycontrast
enhancement,andechosuppressionexploitpropertiesoftemporalreceptivefields,moreprecisely,
delayedinhibition[36,37,100,101,282].Asforechosuppression,itisunderstoodthatitisin
partmonauralandinpartbinauralasvonB´ek´esyandKoenigalreadystatedmorethan50years
[20].agoThemonauralpartofthesuppressioncorrespondstemporallyaswellasfunctionallytothefast
inhibitioninourmodel.Ithasamaximumatsimilartimes(about2ms[108]).Physiologicallyitis
locatedinthecochlearnucleusandhard-wiredneuronallywithoutsignificantvariability[283],just
asinourmodelwherethefastinhibitionisindependentoftheactualechoscenario.Inthemodel
thefastinhibitionsharpensthecontourofthepeakandreducestheexponentiallydecayingtailof
theechobutisnotsufficientforcompleteechosuppression.Similarfunctioninghasbeenfoundin
andtheoreticallyexplainedforthecochlearnucleus[36].
Asforthebinauralpartofechosuppression,thingsaremorecomplicatedbynature.First,in
binauralprocessingmorenucleiareinvolved.Second,theyhavetodealnotonlywiththeinput
fromone,buttwoears.Third,theinvolvednucleireceivemanyefferentfibers,thatis,top-down
controlplaysarelevantrole.Evenifitishardtodisentangletheindividualeffects,theapparent
resultsarethefollowing.First,binauralechosuppressionisslowerthanthemonauralone[283].
Second,itismoreflexible.Moreprecise,ithasbeenshownthatabinauralinhibitorymechanism
suppressesandchangestheconsciousperceptualprocessesofechoesdependingonthesituation
31
SignalEnhancementandEchoSuppression
Figure3.5:Neuronalsetting.AnoisyresponseR(t)incontaininganechoisusedtostimulateaset
ofPoissonneurons[115]A,theso-calleddetectorneurons.Theresultingspikespropagatethrough
delaylineswithdifferentdelaysτ1,..,nandstimulatetheLeaky-Integrate-and-Fire-neuron(LIF)[88]
outputpopulationBthroughthecorrespondingsynapticconnectionstrengthsSc1,...,n.Synaptic
connectionstrengthsanddelaysaregivenbytheoptimalreceptivefieldscontainedinl.Theoutput
Sˆ(t)outoftheneuronsBrepresentstheoriginalsignalwithsuppressedechoandreducednoise.
[221similar].timeComparedcoursetothatthetimestronglycoursedepofendstheonslothewisituation,nhibitioncf.foundFig.in3.4.nature,Foururthermore,modelitshoreduceswsa
thewherethesteepnessflexibilitoftheyecofhotheonset.binauralThisdepsuppressionendencyisupcononthetrolledbysituationthefitstop-dothewneffebiologicalrents.counterpart
theInnaturalsummaryecho,asuppression.neuronalTheimplemenbinauraltationpofarttheofecarchohetypesuppressionmodelwthatorksisveryhighlywell.Itadaptableconnectscouldto
beasiteforfutureresearchtoidentifyandmathematicallydescribecognitiveprocessingsteps.
Discussion3.5Basedonthearchitecturalconceptoferrorminimizationandreceptivefields,wehaveappliedour
frameworkforoptimalmapformationtoaconcretesettingbydeducingaunifyingarchetypemodel
forreal-timeextractionofanacousticsignalfromacorruptedsensoryresponse.Wehavestudied
themodelperformanceinextractingtheoriginalsignalforthreeexemplaryechoscenarios.
Ontheonehand,wehaveshownlinkstocommontechniquesthatsolvetheinverseproblem
[242],applylinearfiltering[141]suchase.g.theWienerfilter[135,228,232,236],pseudo-inverse
techniques,andtheMaximumLikelihoodEstimation[135,228,269].Ontheotherhand,wehave
implementedthearchetypemodelintoaneuronalsettingtocontrasttheoptimalandthebiological
approach.Initially,wehaveposedandansweredthequestionastowhichneuronaldelaysplay
aroleinechosuppression,whataretheirsynapticcouplingstrengths,andwhataretherelevant
timescales(seeFig.3.2and3.4).Wenotethattheresultingsynapticstrengthsofthedelay
linesfollowbiologicalpatternssuchasthetwotemporaldistinctinhibitoryregionsandhintto
cognitiveprocessingsuchastheCliftonEffect[46].Thatis,theauditorysystemadaptsquickly
tocharacteristics(echoesatabout∼10ms)ofaparticularsurrounding.Inmathematicalterms,
thesignalprocessingshowsplasticitybymeansofadaptingquicklytoaspecifictransferfunction
onse).respimpulseom(roMoreover,auditoryobjectformation,bothspeechrecognitionandauditoryspacedevelopment,
areimpactedbythemechanismreferredtoasbinauralechosuppression.Thisisbecausetheslow
inhibitionreducesthesteepnessoftheonsetoftheneuronalresponsetotheechoandauditory
objectformationstronglyreliesontheonsetofacousticsignals[29,292].Theslowinhibition
isthusresponsiblefortheidentificationoftheechoasalocalizableobjectandaconsciousecho
perception.Inthislight,itmakessensethatitisrealizedbinaurallywithstrongtop-downcontrol.
32
SignalEnhancementandEchoSuppression
Figure3.6:Neuronalinputandoutputactivity;forsettings“d”,“r”,and“e”fromlefttoright.
AccordingtoFig.3.5,wefeedthesensoryresponseR(t)inwithanoiselevelof5%(ςt)toa
populationof140detectorneuronsA.Theresultingspiketrains(A−C)propagatethrough
thedelaylineswithdelaysτ1,..,nandstrengthsSc1,...,nandstimulateapopulationof140LIF
outputneuronsB.Thesetofdelaylinesrealizesthederivedoptimalreceptivefieldscontainedin
l.TheupperrowdepictstheactivityofthedetectorneuronsAforthedifferentechofunctions
hd,r,e,thelowerrowthecorrespondingactivitySˆ(t)outoftheoutputneuronsBfortwosetsof
reconstructionparametersσ.Moreprecisely,thelowerrowdepictsthenormalizednumberofspikes
oftheoutputpopulationwithσ=10(D),5(E),and10(F)(graybars).Choosingtherightσ
enhancesthemodelperformanceascomparedtoσ=1(blackbars)thatcorrespondstotheWiener
filter.Obviously,neuronalcharacteristicssuchasthespontaneousrate,hereroughly15Hz,donot
degradetheperformancesincethesignalisextractedreliably.ThethresholdoftheLIFneurons
evenenhancesechosuppression.
Comingbacktothestate-of-the-artalgorithmsfortechnicalde-reverberationandtheirpotential
forimprovement,werealizethatthenaturalapproachismorepowerful.Today’sspeechrecognition
andauditoryobjecttrackingalgorithmscannotcopewithnoisecausedbyreverberationindynamic
scenes.Eventheidentificationofoneacousticsourceinthepresenceofothersisaproblemnot
entirelyovercomebytechnicalsystems.However,arobustauditoryrepresentationaspartofa
multi-modalrepresentationasfoundinanimals[147,261]issupposedtobeanecessaryingredient
forcognitiveapplicationssuchasrobotsandcognitiveworkbenches.Consideringtheeasewith
whichevennotfullydeveloppedbiologicalorganismslike,e.g.,fledglingsmanagetoidentifyand
trackan,untilthenunknown,auditoryobjectsuchastheirmother’svoiceinanewenvironment
itisobviousthatthetwofoldviewonboththebiologicalandthemathematicalsideofthesame
problemdisemboguesintoafruitfulmergeofcultures.
Insummary,ourarchetypemodeluniformlytiesupwiththeknownphysiological,behavioral
mapfindings,formationtechnicalisaaspfeasibleectstoandoltomethoprodsvideofaechocommonconsuppression.vergingThpaterspis,ectivoureonframewsensoryorkofprocesoptimalsing
enablingthemultidisciplinaryapproach–bionics.
33
erceptionPLateral-line4.
Wirk¨onnendieWeltnichterkennen,wiesieist,
sondernnur,wiesief¨urunsist.
erretrandersNoorT
Thelateral-linesystemisauniquemechano-sensoryfacilityofaquaticanimalsthatenables
themnotonlytolocalizeprey,predators,obstacles,andconspecificsbutalsotorecognize
hydrodynamicobjects.Inthischapter,wepresentananalyticapproachthatshowstheeffectof
submergedmovingobject(SMO)shapetolateral-lineperception.Basedontheseresultswe
deriveanexplicitmodelexplaininghowaquaticanimalscandistinguishdifferentlyshapedSMOs.
Ourmodeloriginatesfromthehydrodynamicmultipoleexpansionandusestheunambiguousset
ofmultipolecomponentstoidentifythecorrespondingobject.Furthermore,weshowthatwithin
thenaturalrangeofonefishlengththevelocityfieldcontainsfarmoreinformationthanthat
duetoasimpledipole.Finally,theresultswepresentareeasytointerpretbothneuronallyand
technically,andagreeswellwithavailableneuronal,physiological,andbehavioraldataonthe
system.lateral-line
ductionIntro4.1AllfishandsomeaquaticamphibianssuchastheclawedfrogXenopuspossessauniquesensory
facility,thelateral-linesystem.Thissystemiscomposedofmechanosensoryunitscalledneuromasts
locatedonthetrunkoftheanimal.Theneuromastsconsistofsmallcupulae,gelatinousflags
protrudingintothewater,whicharesensitivetothelocalwatervelocity[21].Theabovementioned
animalsusetheirlateral-linesystemtolocalizepredators,prey,obstacles,orconspecifics.The
laterallineissoeffectivethatitenablesevenblindfishtonavigateefficientlythroughtheir
environmentbymeasuringandanalyzingthepressureandvelocityfieldofthesurroundingwater
[52,111].Moreover,itissupposedthataquaticanimalsnotonlylocalizebutalsoextractand
recognizefeaturesfromthehydrodynamicstructureofthevelocityfieldtodeterminesize,speed,
andpresumablyshapeoftheobjectgeneratingit[39,52].Thiscanbedoneevenindirectly,for
instance,bymeansofextractinginformationfromthevortexeswithintheobject’swake[52].
Thequestionwenowpose,andanswer,iswhetherandhowapassivedetectionsystemsuchas
thelaterallinecanbothlocalizeamovingobjectanddetermineitsshapeinanincompressiblefluid
suchaswater,orairatlowvelocities.Moststudies,bothexperimentalandtheoretical[54,75],
usedavibratingortranslatingsphereasstimulus.Duetothespecialsymmetryofthesphere,the
resultingvelocityfieldisexactlythatofadipole–nomatterwhethervibratingortranslating[171].
Ofcourse,notallobjectsinnaturearespheres;cf.Fig.4.1.Onlyforvibratingbodiescanone
identify[107,142]theinfluenceoftheirshapeontheflowfield,showingareasonabledominanceof
thedipole.Theliteraturelacks,however,ageneralexplanationinamoreglobalcontext.Herewe
showandquantifyhowmuchinformationisavailableinthevelocityfield,howonecanextractit
andtowhatextentindependenceuponthedistance.
35
ineLateral-l4.erceptionP
Figure4.1:Submergedmovingobjects(SMOs)appearinwidelyvaryingshapes.Furthermore,
aquaticstimulimaybutneednotmoveatall;forexample,vortexstructuresthataregeneratedin
thewakeofaswimmingfishorattheendofthefins[52].Thesestimuliimprintinformationonthe
flowlateral-linefieldthatorganscanisbemoreaddifiedoutbdueytothequanlateral-linetitiessuchorgans.astheInSMO’saddition,ptositheion,inputmotion,measuredandatshapthee.
Thebodyofthedetectingfishalsoinfluencestheflowfield,forinstance,bymeansofdamping
(FromNOAA’sHistoricFisheriesCollection;IDs:fish3078andfish3039)
Moreinparticular,wefirstderiveanalyticalequationsshowingthatshapeinformationplays
anessentialroleaffectingthevelocityprofileatthelaterallinewhichisthenaturalstimulusto
inthetermsmecofahanosensorymathematicalsystem.Second,descriptionweinthreedemonstratedimensionshowhisydropossibledynamicbyobjectmeanscofamharacterizationultipole
apreexpansiondator,can(ME).readInoutaddition,thevweloecitshoywfieldhowandanaquaticreconstructanimal,theshapcalledeofasdetectingtimulus,animalviz.,(DaA),subsmuchergedas
movingobject(SMO),suchasprey.Finally,wegivelinesofevidencethattherearebiologically
therelevanrangetuplimitstoofwhichstransmittinghapeandinformationextractingisimportaninformationt.The–forlatterboth,hasthenotbeencapabilityexploredtoloupcalizetonoandw.
4.2Analyticdescriptionofshape
Asexplainedabove,disturbancesofthesurroundingwaterformthenaturalinputtothissensory
system.Theinputmay,thoughneednot,begeneratedbysubmergedmovingobjects(SMOs)and
dependsontheirposition,motion,andshape.Thebodyofthedetectingfishalsoinfluencesthe
flowInfieldthis,cforhapterinstance,wefobycusonmeansandofisolatedampingthewaterinfluenceofdisturbances,theSMOviz.,shapthee.signal;Tocf.outlineFig.the4.1.basic
poideawoferfulthetwpresento-dimensionalsection–anmethodsanalyticalbasedonthedescriptioncomplexoftheforminfluenceulationofofstimpotenulustialshapfloew–devweeloprecalled
forairshipdesign[270].Potentialflowisavalidhydrodynamicdescriptionofthelateral-line
perceptionalthoughitneglectseffectsofviscosity[95,111].Acomplexpotentialflowdescription
ofthree-dimensionalhydrodynamicsettings–includingevenrotational-symmetricbodies–is
ellipticallyanalyticallyshapnotepdossibleprofiles(seewithinApptheendix(x,yB).)Wplane.eInthereforedoingso,reducewethegainobjectsanalytictoalaccesscircularlytoandthe
essentialeffectsofobjectshapeontheflowfield.
36
4.Lateral-linePerception
a=Figure10−43.2:,2.5Flo,3w.5;aroundtheradiusthreer=differen4,andtlyV0shap=ed1.Weprofiles.indicateFromthelefttoprofilerighbyt:grashapy.eOneparametercanseeho(4.2)w
theprofilebecomesellipticandthestreamlinescorrespondingtotheinputtothelaterallineadapt
accordinglytoobjectshape.
rmulationfoMathematical4.2.1ThecomplexvelocitypotentialF(z)candhencethecomplexvelocityw(z)c:=dF(z)c/dzwith
x+iy=:z∈C,aroundacircularprofile[218]isgivenby
F(z)c:=V0z+r2(4.1)
zthewhereJoukV0owskidenotesthetransformationmovingspζ(eedz):=andzr+athe2/zradiuswithofshaptheecircularparameterprofile.a∈InR[the218],nexti.e.,step,wconformaleuse
mapping,toobtaintheflowfieldwe(z)aroundanellipticallyshapedprofile.Herethesuperscripts
ctheandfloewreferfieldwtoeincirculartheζand-planebutelliptical,intherespectivoriginalely.planeInconoftrtheasttosphere,thetheusualzwa-plane.y,weThwillatis,notwestudycan
identifythecoordinatesandthuscomparetheflowofthecircularandellipticalprofile,
edFdzr2a2−1
w(z)=dzdζ=V01−z21−z2.(4.2)
wc(z)ddζz
Inthiswaywehavederivedthecomplexvelocitywe(z)oftheflowaroundanarbitraryelliptic
eandshape.theFuvelocitrthermore,yfieldwveecan=(ve,computeve)byallinmeansterestingof(4.2)quanandtitiesz=xsuc+hiyas.theAllvelothesecityptermsotenaretialΦlisted(x,yin)
Table4.1.Thetransformationxyfromtheflowaroundcircularprofilestotheflowaroundelliptical
onescanbedescribedbyonlyoneparametera∈R.Asa→0,allellipticalquantities(seeTable
4.1)convergetothecorrespondingquantitiesofthecircularflow.
Wesummarizethesethoughtsbystatingthatthecircularflowisonlyaspecialcaseofthe
moregeneralellipticflow(4.2).Asaconsequence,weobviouslycannotexpectaprioristatements
derivedfromcircularscenarios,i.e.,thespecialcase,toholdgenerally–ifatall.
eshapofEffects4.2.2Havingderivedageneralexpression(4.2)fortheflowaroundanellipticallyshapedprofile,wenow
exhibitfundamentalmodificationsofthevelocityprofileasitismeasuredbythelaterallinedue
tostimulusshape.Inthepresentliterature,onedistinguishestwoclassesoftheoreticalmodels
thatexplainthedetectioncapabilityofthelateral-linesystem[54,75,95,251].Modelsofthefirst
thecategoryfish’sboexplaindy.Thethepseerfcondormanceclassofofmothedelslateral-lineexploitschsystemaracbteriysticusingpointhetssucwholehasvelocitextremyaprofileandzerosalong
37
erceptionPineLateral-l4.
Figure4.3:TherelativequadraticerrorΔofthevelocity-profileextremaatthelaterallinexEin
dependenceuponthedistanced/randshapeparametera/rnormalizedbytheradiusr.Thegraph
showsΔ:=[xE(0)−xE(a)]2/xE(0)2ford/r=0.5,1,1.5.Fordistancesdthatarecomparable
totheSMOsize(radiusr),viz.,inmathematicalterms(d≈r⇔d/r≈1),thepositionsofthe
extremaxEclearlychangewiththeshapeparametera.Presentmodelsignoringshapeeffects
calculatingthedistancedfromalinearrelationd∼xEthereforeoverestimatetherealdistance
beyondd≈r.
ofthevelocityprofilealongthelaterallinesincethesepointsaresupposedtobeeasilyaccessible
.neuronallyWefirstanalyzethelattercategory.Knowingtheorientation,i.e,thedirectionofsensitivity
g=realizedg(x,byy)andmeansposofivtiong(rr)==r(dΦx,/ydg).ofIntheotherwlateral-lineords,weorgans,canevcalculateerythelateral-linewatervelogeometrycityincanevberye
directiongandatanypointroftheflowfield.Here,foreaseofcomparisonwithcommonscenarios
inthethedetectingliterature,animal.weInstudythisansetup,SMOcenthevteredelocitaty(0,profile0)ispassingidenticalalongwiththevx-axisandatyadenotesdistanceyexactlyto
xtheWedistancestartbdybresetwcalineengtheourcentersexpressionoftheforvSMOxbyandmeanstheoftrunknormalizinglateralline.allquantitiesbytheSMO
size,i.e.,theradius.Thisresults,forinstanceina→(a/r)r=ar.Thenormalizationmakesv
explicitlyindependentofr.Computingthex-valuesxEoftheextrema(skippingxE=0)bymeansx
ofcomputing∂vx/∂x|x=xE=!0givesxE(a)=±a2+dd±2√a2+d21/2.Obviously,the
extremadependontheshape;inmathematicalterms,ontheshapeparametera∈[0,1].Onlyin
thecaseofacircularprofile(a→0)dotheextremaencodethepuredistancexE(0)∼d.Common
tomothedelszerosexplaind∼the(xloE1−calizationxE2)[p54,75,erformance95].oftheWithoutlateral-linetakingthesystemshapbeyintomeansaccounofat,linearthepresenrelationt
modelsthereforeoverestimatetherealdistance.
detectingAtthesamelateral-linetimesystemtheaseffectwellofasshaptheeSMOalsobdepodyendslengthonthethatdistancedirectlybetrelatesweentotheSMOandparameterthe
rangeradiusrdefinedintheastheequations.distanceAsupwetocantheseeSMOinbFig.ody4.3,lengththeandinfluvenceanishesofforshapelargerdominatesdistancesthednearb1.y
Nextweturntotheotherclassofmodelsthatnotonlyexploitparticularpointsbutthewhole
pvelootencitytialfieldofatellipttheicallateralprofilesline.Fe(Tzo)inanalyzeaTaylortheseriesinfluencearounofdshapa=e,0,weexpandthecomplexvelocity
Fe(z)=V0z+rz+rz3V0−Vz0a2+O[a]4.(4.3)
22
Fc(z)
Throughthesubstitutionc:=r2/a−awecantransform(4.3)intoapowerseriesinz,
Fe(z)=V0z+V0ca+a3+a5+O[a]7.(4.4)
zzz
38
4.Lateral-linePerception
Table4.1:Hydrodynamicetoolbox:Comparison2oftheflowaroundcircular(denotedbyc)and
theellipticalflowaround(denotedanbyellip)ticalprofiles.profilebDefiningymeancs:=ofr/aonly−a,oneweshapcanederiveparametersimplea.equationsdescribing
Fc(z)=V0z+rz
Complexvelocityp2otential
Fe(z)=V0z+2c[ln(z+a)−ln(z−a)]
Hydrodynamicpotential
22Φc(x,y)=V0x+x2r+2xy2
Φe(x,y)=V0x+4cln((xx−+aa))2++yy2
Velocityoftheflowinx-direction
vxc(x,y)=V01+x2r+2y2−(x22r+2yx22)2
vxe(x,y)=V01+2[(xc+(xa)+2a+)y2]−2[(xc−(xa)−2a+)y2]
Velocityoftheflowiny-direction
vyc(x,y)=−V0(x22r+2yy2x)2
22vye(x,y)=−V0a4−2a2(2(x2r−−y2a))+(yxx2+y2)2
Equation(4.4)showsmoreclearlythan(4.3)thattheshapeparameteraisthenaturalexpansion
parameterforthecomplexmultipoleseriescharacterizingshape.Wewillexploitthederivedfinding
thatmultipolescanbeusedtodefineandcharacterizehydrodynamicobjectshapelateron.
Thedifferencebetweenthefloworiginatingfromacircularandanellipticallyshapedprofile,
andthustheimpactofshape,canbeexpressedtofirstorderby
22ΔF(z)=[Fe(z)−Fc(z)]=−arV0+O[ar]3(4.5)
zwherea=a/risagainnormalizedandrangesbetween0and1.Itiseasytoseethatthecomplex
velocitydifferenceΔw(z)=dΔF(z)/dzandthusthemeasuredvelocitydifferencescalesas(r/z)2.
ThebodylengthoftheSMO(≈r)thereforedefinesanaturallengthscaleuptowhichtheeffectof
shapecontributessubstantiallytotheflowfield.Withinthisrangethedipoleapproximationis
onlyavaliddescriptionforcircularshapes.
Insummary,shapeshiftsthepositionofcharacteristicpointsofthevelocityprofileandalso
modifiesitasawhole.BotheffectsdependontherelationbetweendistanceandSMOsize.Since
underpotentialflowconditionwehavep∼vv[95],theSMOshapeaffectspaccordinglyand
thusalsothecanallateral-lineprocessing.Moreoverwehavegivenananalyticallineofevidence
thatthenaturalrangeuptowhichinformationaboutshapeistransmittedistheSMObodylength.
Havingdiscussedreallysimplebutneverthelessnon-trivialshapes,wenowwanttogivean
outlookonpossibleextensionsofourmodel(4.2).Itisstraightfrowardtoadapttheequations
inTable4.1tomorefish-likeshapes.Substitutingz→(z+b)withb∈RintheJoukowski
transformationleadstomorecomplicatedbutricherexpressionswheretheadditionalshape
parameterbinreturnallowsformorerealisticshapes.Wehavedepictedsuchshapesandresulting
velocityprofilesinFig4.4.
Inthenextsection,however,weturntothree-dimensionalshapes,theirinfluenceonlateral-line
perception,and,moreimportantly,howthelaterallinecanrecognizesuchdifferentlyshapedSMOs.
39
erceptionPineLateral-l4.
Figure4.4:Effectofthesecondshapeparameterb.Thegraphsdepicttheflowfieldaroundthree
differentprofileswithvaryingthesecondshapeparameterb=0,−0.1,−0.3(fromlefttoright).
TheradiusoftheSMOisr=1andtheshapeparametera=0.5from(4.2);bothrandaare
constant.Theprofileitselfisindicatedbythegrayarea.Duetovariationofbtheprofilebecomes
moredrop-shapedandthestreamlinesadaptaccordingly.
recognitionobjectdynamicHydro4.3mInultipthisolespart,iswecapablewanttotopicdescribkeupthetheinfluencefindingsooffSMO(4.4),shapnamee,lyandthatderivaeseriesbothofahydromathematicaldynamic
definitionanddescriptionofhydrodynamicobjects.Inaddition,wederiveanoptimalobject
inthreerecognitioncoordinates.algorithmInthatdoingcanso,lowcalizeeareandabletoreconstructpredictptheositionandcapabilitiesshapofeoftheSMOlateral-linesimultaneouslysystem.
4.3.1Definitionofhydrodynamicobjects
Aswediscussedintheprevioussection,avelocityfieldrepresentsanadequateandnaturalstimulus
tothelateral-linesystemandcanbedescribedbyamultipoleexpansion(4.4).Now,weleavethe
two-dimensionalspaceandfocusonthreedimensions.Becausetherelevantfluiddynamicsarewell
realdescribveloedcitbyyptheotentialEulerv(r)=equation−[Φ(95r])w[e171tak]efortheathree-startingdpimeoinnst.ionalUsingvelothecityrealfieldsphericalderivedfromharmonicsthe
YlRmandthesphericalmultipolemomentsq=(...,qlm,...),wecanexpandthevelocitypotentialΦ,
llR∞∞Φ=qlmΦlm=qlm1Ylml(θ+1,ϕ)(4.6)
l=1m=−ll=1m=−l2l+1r
describForedthesakthrougheofcosimplicitordinatesyweς∈fo[0cus,α]onandηrotationally∈[0,2π],symmetricalbodieswhosesurfaceScanbe
−1/3
ςπ√2γsin(α/2)sin(ς)cos(η)
S(ς,η)=Ncosα3×√2γsin(−1/α/32)sin(ς)sin(η)(4.7)
γ2/3cos(ς)
Thethatvsurfaceortexstructuparametersresαcan∈be(0,πdesc/2]ribandedbγy∈(0means,∞)ofadeterminemultipoletheshapexpanesofionthe(ME)surface.aswellWe[171note].
BecauseSMOssuchasfishleavecharacteristicvorticesintheirwake,thefindingswederivein
thisandthefollowingsectionsarealsoapplicableforidentifyingSMOsindirectlybymeansof
characteristicvorticesintheirwake.
BecauseoftheEulerequation,onlytheNeumannboundaryconditionv∙nS=0isrealizable
wherenSdenotesanormalvectortothesurfaceS.Intermsofsphericalcoordinateswecalculate
40
4.Lateral-linePerception
Figure4.5:Differentshapeparametersresultindistinguishablesetsofmultipolecomponents.We
depictanapproximationforonlythreemultipolemomentsl≤k=3in(4.8).TheSMOsurface
parameterαincreasesalongtheverticalaxisandtheSMOsurfaceparameterγalongthehorizontal
one.Wehavecalculatedthesetofmultipolesqˆthrougharasterof30,000randomlydistributed
positionsoneachmovingobject’ssurfaceandusedobjectswithlengthsofabout5cmanda
matchingmovingspeedv∞=0.01m/s.Eachvolumeisnormalizedtothevolumeoftheunitsphere.
Itisthereforefairtosaythatthedipoleq10staysalmostconstantwhilethequadrupoleq20varies
accordingtoαandtheoctupoleq30accordingtoγ.Therelativeerror(k):=|Tqˆ−w|/|w|on
theSMOsurfaceSascalculatedforrotationallyinvariantellipticbodiesgives(k)<0.4fork=3
andγ<2.Itconvergesslowlytoabout0.15fork→∞.Forγ2amultipoleexpansioncannot
describethecorrectvelocityfield.This,however,isnottoorestrictive.Forexample,thefishin
Fig.4.1canbedescribedbyγ≈1(leftfish)andγ≈2(rightfish).
thecomponentsofv(r)=−Φ(r)=(vr,vθ,vϕ)atrthroughcoefficientsarlm,etc.,sothat
∞l∞l∞l
vr=l=1m=−lqlmarlm,vθ=l=1m=−lqlmaθlm,andvϕ=l=1m=−lqlmaϕlm.(4.8)
Thegeometry-depequationsending(4.8)cancobeefficienwrittentsarlinm.Tmatrixoexplformicitlyv=AcalculateqwiththethesinglematrixmAultipconoletainingmomenthetsqlonlym,
weuniformhavesptoeedspvecifyofthethebSMOoundaryintheconditionstationarywithrespframeectoftonSreference.and,forThethebsakoundaryeofsimplicitconditiony,thenthe
∞readsv(rS)∙nS=nS∙v∞=nSTA(rS)qwithrS∈S.(4.9)
theThevelocitinfluenceyfielofdtheonlymupultiptoaolesincertain(4.8)numbdecreaseserkandwithsetqlmincreasing=0lfor;lsee>k(4.6)..WeHencearethwuseexpanddealing
withanapproximatedsetofmultipoles,denotedbyqˆ.Tofindanappropriateapproximation
wassoetakciatedesevsurferalacpeositionsnormals1n≤i,i≤thenmatrixrandomlyAi,anddistributedwriteitonlineanbySMOlineinsurfacetotheandtransfercalculatematrixthe
41
ineLateral-l4.erceptionP
T:=(...,niTAi,...).Theindexiofthevelocitycomponentswi:=v∞∙niofthevectorw
spcorrespeedvondsoftotheniSMOandleadsthereforetolabincreasingelsthevsamealuespofosition.thecorrespHere,weondingnotemthatultipanolemomenincreasingts.moInvingthe
∞nextsection,thiswillbecomeimportantinthecontextofthedetectororganssignal-to-noiseratio.
TocalculatethemultipoleswesimplysolvethelinearequationTqˆ=w.Themultipolemoments
followfromqˆ=(TTT)−1TTw[228].Thesolutionapproximatesthemultipolecoefficientsuptoa
givenkoptimallyinthesenseofminimalquadraticerror.Introducingacharacteristiclengthscale
λ,saythebody-lengthoftheSMO,intoequation(4.6)weget
v(r)=qlm(λ/r)l+2flm(θ,ϕ)(4.10)
,mlwhereflm(θ,ϕ)arefunctionsdependingonlyontheangularcoordinatesθandϕandcanbe
calculatedbymeansof(4.6).Indoingso,weobtaindimensionlessmultipolemoments.Onthe
onehandwecancomparethemindependentlyoftheSMOsize.Ontheotherhandweseefrom
v(r)=l,m(λ/r)l+2qlmflm(θ,ϕ)howtheinfluenceofeachmultipolevariesindependenceupon
thetellsussizeλthatofthetheshapSMOeofandthetheSMOisdistanceonlytoimptheDortanAtlabiftheledebsizeyλtheandradialthecodistanceordinaterarer.ofThistheresultsame
willnotmagnitude.transmitItisane.g.yobshapeviousthatinformationplankton,throughwhicthehisflowtenfieldtoh–andundredinthistimescasesmallershapethantheinformationDA,
isnotneededeither.Thisresultisinperfectcorrespondencywithwhatwehavederivedpreviously
startingfromdifferentequationsinthecomplextwo-dimensionalspace;cf.Fig.4.3.
However,asstatedbefore,inschooling,matefinding,orpredator-preybehaviorobjectinforma-
tioncanmakeadifference,andasshowninFig.4.5,differentlyshapedobjectsarerepresentedby
differentsetsofmultipolemoments.Consequently,multipolesareabletoquantitativelydefineand
describehydrodynamicobjectsaswehavedemonstratedforthefirsttime[251]–asfarasweknow.
4.3.2Objectrecognitionperformance
How,then,canaquaticanimalssuchasfishreconstructtheabovemultipolemomentsfromwater
velocitymeasuredthroughtheirlateral-linesystem?Ofcoursetheanimalhastofulfillthistask
undertheinfluenceofomnipresentnoiseandthelimitationsofitsneuronalsystem.
Ourreconstructionmodelforthree-dimensionalshaperecognitionisbasedonamaximum-
likelihoodestimator[276]andtheframeworkofoptimalmapformationthatwehaveintroduced
inChap.2.Thatis,wearelookingforthemultipolesqwiththehighestprobabilitygiventhe
measuredvelocitieswontheDAbody.Sotospeak,thisestimatormaximizestheconditional
probabilityp(q,r0|w)fortheSMOcenterofmassr0,whichhasalsotobedeterminedbymeansof
theinformationavailablethroughtheDAsuperficialneuromasts,thewatervelocities.Thevelocity
sizewiatorganiisgivenby
wi=T(r0,ri)q+ni.(4.11)
WenotethatthetransfermatrixT(r0,ri)dependsontheSMOpositionr0andtheposition
riofthelaterallineorgani(DA).Asintroducedinthefirstchapter,thetransfermatrixmaps
thesignalontothedetectors.InthiscaseT(r0,ri)projectsthemultipolesqlmontothevelocity
componentswimeasuredatlateral-lineorgani.NoiseismodeledbyaddingindependentGaussian
randomvariablesniwithmean0andstandarddeviationσntothewi(4.11).Allpositionsr0areof
equalprobability.WethereforetakeaGaussianprobabilitydistributionp(q,r0)∼exp(−q2/2σq2).
InsteadofmaximizingthecombinedprobabilitybasedonBayeslaw,wemaximizeitslogarithm
L(q,r0):=ln[p(q,r0|w)].Definingthedimensionlessparameterσ:=σn/σq,wejusthaveto
maximizeL(q,r0)=−[w−T(r0,ri)q]2+σ2q2(4.12)
42
4.Lateral-linePerception
Figure4.6:Neuronalimplementationof(4.14)isstraightforward.Inafirststep(A)→(B)the
vmeloultipcitiesolewicompattheonentsqˆlmneuromasts(r0)((BA)).areThiscalculatedcorrespforondstodifferenanettwporkositionsofr0synapticfromtheconnectionsmeasuredbetwweenater
thelcomputedateralfromlineandtheenthetriescenoftralthenervrighoust-handsystem.sideTheof(4.14)strengthorofcanthebelearnedindividualneuronallyconnections[74].canInbae
secondconnections;step(Bcf.)→(4.12).(C),LThe(q,mrax0)imisumL(calculatedqˆ,rˆ0)fromindicatestheqˆrthe0,whiccorrecthpcanositionalsobofethedoneSMObyandfeedforwselectsard
theoptimalqˆr0fromstep(C)→(B);cf.(4.13).
thewithlikrespelihoectodtoL(theqˆ,rˆm0)ultipattheole-momencorrectlytvectorestimatedqandpr0.ositionTherˆ0isnecessaryconditionforamaximumof
∂L(q,rˆ0)|q=qˆ=TTT+σ2q−TTw|q=qˆ=0,(4.13)
q∂whichleadstoalinearsystemofequationssolvablebymeansofthepseudo-inversetechnique[228],
qˆ=TTT+σ2−1TTw.(4.14)
AWeneuronestimatealtheimplemensetofmtationultipofolesthqˆesegivenonlycalculationstheismeasuredstraighvtforwelocitiesardwandatcanthebedonelateral-lineeasilyorbgans.y
neuronalhardware;seeFig.4.6.Thusthepositionandtheappropriatemultipolemomentscanbe
.neuronallycalculatedTotestwhetheritispossibleforaquaticanimalstodetermineSMOpositionandshapeusing
theabovemethod,wehaveappliednoisetotheinputsignalwi.Wehaveusedσn=10−4m/sas
standarddeviationofthenoise,whichcorrespondstothevelocitythresholdofXenopus’lateral
lineorgan[21].Furthermore,wedefineasignal-to-noiseratio,SNR:=v∞/σn.Plausiblemaxima
ofthemovingspeedofSMOsrangebetweenonebody-lengthpersecondandzero.Tobefair,in
theexamplesinthefigureswehaveusedavelocityaslowasv=1cm/scorrespondingtoonly
1/5to1/10ofthetypicalmaximummovingspeed.Thatistosa∞y,wehavedealtwithaworse-case
scenariosinceincaseofhighermovingspeedv∞thereconstructionperformanceisevenbetter
thanpresented.Anotheraspectregardinglateral-lineperformanceisthenumberofdetectororgans.
Withinabiologicalrangetheinfluenceoftheexactnumberoflateral-lineorgansismarginal.We
butshowstainysFig.almos4.9tthatconstanthetfordiffreconstructionerentnasumwbellersasoftheloneuromasts.calizationOferrorcourse,increasestaking,withfortheinstance,distanceten
timesmoreneuromastscancompensateahigherSNRvalue–butonlywithinmarginalranges.In
summaryAnother,theissueSNRis,thedominatesmoremtheultippoleswerformanceeuseasthemorecomparedptoositionsthearenumofbherighofprobabilitneuromasts.ybecause
onethehasshapemoreestimateparametersimprovtoes.fitWtheecancmeasuredhoosewian’s.Thusappropriatetheσestimatedforapcertainositionktogetscompambiguousensatebutthis
43
ineLateral-l4.erceptionP
Figure4.7:Thelateralline(A,grayline)iscenteredat(0,0).Thedistancebetweentheobject
(SMO,grayarea)centerofmassr0andthelateralline(DA)isjustbelowthelengthofthelateral
line.Usingthedipoleterm(k=1)only,wecaneasilyreconstructthepositionrˆ0(A,cloudof
mblacultipkpoleointsmomenaroundts(rk0=),3a)asresultanefficienconsistenttwiminimalththatmoofdel,Fourranoschmethoetdal.[allo74ws].Ifaweproptakereonlyestimatethreeof
bothposition(A,smallopencircles)andform,cf.Fig.4.8–eventhoughmorepositionsarenow
likelytooccur.Forbothcases,40reconstructionshavebeendepictedwithσn=10−4m/smoderate
SmapNRof:=pv∞osition/σn=100estimation.andIt500showsneuromasts.noamBbiguity;depictscf.logFig.[L4.(qˆ,6C.r0)]correspondingtotheneuronal
effectandenableafaithfullocalization;fordetailsofhowtotuneσconsiderChap.2.InFig.4.7
weshowtheabilityofourmethodtoestimatetheSMOpositionrˆ0.
Followingtheansatzoftheneuronalmodel(Fig.4.6)wehavecalculatedthemultipolemoments
attheestimatedposition,i.e.,notonlyatthecorrectpositionbutalsoatrˆ0+δrwhereδraccounts
forthecontinuityofrealspace,alimitednumberofmapneuronsaswellasaslightlocalization
uncertainty(Fig.4.7).Ourmodelcanreconstructqˆr0evenundernoisyconditionsandatslightly
wrongpositions(Fig.4.8);forexample,thefishbodiesofFig.4.1(seealsoFig.4.5)canbe
recognizedanddistinguished.Thusobjectlocalizationaswellasobjectrecognitionbasedonthe
estimationofmultipolesispossiblesimultaneously.
Inthesubsequentlineswediscussthelimitationsoftheproposedmethodandlikewisethe
theoreticallimitationofaquaticanimals’abilityofobjectrecognition.Tocomparethequalityof
differentmultipoleestimatesqˆatdifferentdistancesd,wedefineδq:=(qˆlm−qˆlm)2/qˆlm
asqualitymeasure.Hereδqlabelsthenormalizedstandarddeviationduetowronglyestimated
positionsrˆ+δrandnoise.Forsmallδqtheestimationerrorinqlmissmalland,inspiteofaslightly
wrongposition,theshapeparameterγcanberecognized.Forvaluesδq>1ashapereconstruction
basedontheestimatedqˆlmisimpossible.
Theresultsofournumericalexperiments,thatis,thelateral-lineperformanceourmodelpredicts,
supportthepreviousanalyticalfindingsasfollows.Moreprecisely,fortheoctupoleq30thecritical
distancewheretheerrorδqstartstogrowextremelyfastistheSMOsizeitself.Forlocalizationthe
criticaldistancecanbetakenasthelengthoftheDAlateral-linesystem,sayafishlength,which
isincaseofapredatorhuntingforpreylargerthanthecriticallengthofshapereconstruction
4.9).(Fig.
44
4.Lateral-linePerception
qˆ30Figure(I−4.8:L)normalizedReconstructedbythedipoledipqˆole10(Astrength−D),q10forquadrupα=oleπ/qˆ220(andEv−Hariable),andshapoesctup1ole≤γ≤strength2at
withdifferentnoiseatdistancesthed.mostThelikelygraypositioncolored(lighregiontgrey)depictsandtheatvslighaluetlyofincorrectlyestimatedmultipestimatedolepstrengthsositions
(darkgrey)asexplainedinthemaintextandcorrespondingtoFig.4.7.Sincewehaveusedan
approximationwithonlythreecomponents,thestrengthsofthemultipolesdifferfromtherealqlm
(blacdistinguishkline).differenRemarktlyablyshap,qed30isobanjects.approWeseeximately(A,B,linearE,F,Ifunction,J)ofthatγforandthsmallusdialmoststancesbsufficesetweento
ME.SMOIfdandDAapproacdifferenhestlySMOshapsizeed(5bocm),diesthecanbereconstructiondistinguished,startsevengettinginspiteblurredofnoise(B,Cand,F,G,Jtruncated,K).
Atlargerdistances(D,H,L)nostrongcorrelationbetweenqˆ10,qˆ20orqˆ30andγcanbefound,and
thusnoshapecanberecovered.NeverthelessthetwofishinFig.4.1with,say,γ=1andγ=2
canbedistinguishedatleastatd=3cm(A,E,andI)andd=4cm(B,F,andJ)bymeans
ofestimatingthewholesetofmultipolemomentsoronlythestrengthofqˆ30.Inbothcasesqˆ30
discriminatesγ=1andγ=2.
Insummary,ourmultipoleexpansion(ME)methodcanquantifystimuluscharacteristicsby
meansofreconstructingthemultipolemomentsqlm.Becausetheflowfieldneednotbegenerated
byasubmergedmovingobjectalone,themethodcanalsobeappliedtocharacterizecomposite
depictedsituationsinsucFig,has4.8schoandolingFig.orv4.9ortexwehavestructuresansweredfoundtheinthecompwakeellingofafish.question,Inpresenwhethertingatheresdetectingults
animal(DA)orartificialsystemcanlocalizeandrecognizeaSMOgivenonlythemeasuredwater
velocitiesattheDAlateral-lineorgansand,andifso,howwell.Thatis,theMEisusefulonlyif
theSMO,ordistanceshorbteetr.weentheOtherwiseobjectthe(SMEMO)reducesandtodetectoritsdip(DoleA)isapprosimplification.ximatelyofthesamesizeasthe
45
erceptionPineLateral-l4.
Figure4.9:Thetwographsdepicttherelativereconstructionerrorδq30(dashedline)oftheoctupole
momentq30asdefinedinthemaintextfordifferentnumbersofneuromasts.Theyshowthat
fordistancesdlargerthanamovingobject’ssize(5cm)multipolereconstructionisnotpossible.
Thesolidlinedepictsthelocalizationerror||rˆ0−r0||,whichstartstogrowfastatadistance
comparabletothelengthofthedetectinganimal(10cm).Thisagreeswellwithexperimental
findingswherelocalizationperformancestartsdecreasingatdistancesbeyondthefishbodysize
[54].Fordiscriminationtasks,fishhavetobeveryclosetotheobjectunderinvestigation[39].The
leftgraphshowstheerrorsemploying250(left)and500(right)neuromasts.Asexpected,inboth
plotserrorsincreasewithdbutthedifferenceduetothenumberofneuromasts(withinbiological
marginal.isrange)
okoutloandDiscussion4.4Inthischapter,wehavefocusedontwomajoraspectsoflateral-lineperformance.First,nomatter
whethertheanimal’scapabilitytolocalizeprey,predators,conspecifics,orobstaclesisbasedonthe
processingofcharacteristicpointsofthevelocityprofileorthewholevelocityfield,theSMOshape
playsanessentialrole[253].Assumingthatpotentialflowisanadequatetoolforapproximating
thecorrespondinghydrodynamics,wehavequantifiedanalyticallytheeffectofshapetogainfurther
insightsintomechanosensoryprocessing.Inadditionandforthefirsttime,wehavepointedoutthat
differentlyshapedobjectsresultindifferentboundaryconditionsforthehydrodynamicflowfield
andthuscausedistinguishablemultipolemomentsqlm.Wehavedevelopedamultipoleexpansion
method(ME)thatpredictsthepossibleobjectrecognitionperformanceofthemechanosensory
sensorysystem–thelateralline.
Second,inthepresentliteraturethereisanaturalcharacteristiclength-scaleuptowhichthe
animalcanlocalizeobjects,viz.,itsbodylength,ormoreprecisely,thelengthofitstrunklateral
line[95,75].Here,wehavegivendifferentlinesofevidencethatthereisasecondlengthscale
withinwhichshapeinformationistransmitted,namely,thebodylengthoftheSMOtobedetected.
IfthedistancebetweentheDAandtheSMOisbelowthatlengthscaleeventhesimplestobjects
transmitmeasurableshapeinformation.Thisinformationcanbeusednotonlytolocalizebuteven
torecognizehydrodynamicobjects[251].
46
4.Lateral-linePerception
Promisingstepsforfurtherenlighteningthisinterestingsensorysystemaremanifold.Inconcreto,
wesuggesttoexperimentallytesttheabilityofthelaterallinetorecognizeshapeindifferent
settingstoevaluatethepredictedrecognitionperformance.Onecouldclarify,forinstance,the
numberofmultipolesthatisnecessarytorecognizeshapeinnaturalconditions.Anotherinteresting
aspectwouldbetoanalyzewhetherthesensorysystemis“blind”whenitcomestodistinguish
SMOswithspecialsymmetries.Moreover,thepossibleexperimentalsetupsmaynotbelimitedto
thelateralline,viz.,hydrodynamics.Air-dependentmechanosensorysystemssuchasthesensory
systemofcricketscanalsobeconsidered.
Onthetheoreticalside,itwouldbeveryinterestingtoextendourmodeltothetimedomain.
Thatis,ontheonehandmodelsthatenhancethepresentMEmodelshoulddealwithatemporal
varyingmultipolepatternq(t)tofaithfullyidentifyhydrodynamicobjects.Namely,themoving
bodyorthevorticesinthewakeofswimmingfishproducespecifichydrodynamicstructuresthat
aretime-dependent,whichistrueforthecorrespondingmultipolesaswell.Presumably,the
time-dependentmultipolesareanimalspecific.Ontheotherhand,itispossibletoextendthe
presentMEmodeltonotonlyreconstructthemostprobableSMOpositionatonespecificpointin
timebuteventoestimatethemostprobableSMOtrajectory.Giventhetrajectoryandassumptions
onthemovingobjectsuchasitsinertmassitiseasytopredictitsfuturepositions.Thedipole
momentincludestheactualmovingdirectionandspeedoftheSMOandwouldthereforealready
with.starttosufficeInsummary,ourobject-recognitionmodelagreeswellwithbiologicalfindingsandprovidesa
theoreticalunderstandingofhydrodynamicobjectperceptionthroughthelateralline.Furthermore,
wenowunderstandthefundamentalrestrictionsforany(sensory)evaluationoflateral-linedata.
Finally,thesefindingscanalsobeappliedtobiomimetics[289];e.g.,toimprovepassivenaval
ms.systevigationna
Figure4.10:Sometimes,itmakesadifferencetoknowwhatiswhere.Frommyposterpresentedat
the2009Sloan-SwartzSummerSchoolatHarvardUniversity.
47
ihrDieeGeWahrheitgnersterbtrienumphiertnuraus.nie,
5.InfraredSystemofSnakes
kPlancMax
Somesnakespossessaninfrareddetectionsystemthatisusedtocreateaheat-imageoftheir
chaenvironment.racteristicsInofthissensorychapter,en-andwedecoexploding,retheandinvolvedaddressthephysicsquesoftionsignalofhowmapping,theinfratheredneuronalsystem
cancreateafunctionalrepresentationofthethermalpropertiesoftheenvironment.
ductionIntro5.1Twogroupsofsnakes,pitvipersandboids,possessasensorysystemtodetectinfrared(IR)
radiation[198]whichallowsthemtoperceiveatwo-dimensionalimageoftheheatdistributionin
theirsurroundings.Thedetectionsystemconsistsofasetofcavitiescalledpitorgans.Inpitvipers,
apitorganislocatedoneachsideofthesnake’sheadneartheeyes;cf.Fig.5.1A.Suspendedin
eachcavityisaheatdetectingmembrane,whichissensitivetomKtemperaturedifferences[33,63].
TheopticalprincipleunderlyingthedetectionofIRradiationisthatofapinholecamerawithlarge
aperture;cf.Fig.5.1B.
Radiationenteringthroughthepitholehitsthemembraneatacertainspotdependingonthe
sourcedirection;seeFig.5.2A.Theresultingtemperaturechangeatthisspotisdetectedbyheat
Figurecourtesy5.1:of(A)GuidoHeadWofaesthoff;pitviperadaptedwithfromnostril,[252].(large)(B)pitApihole,tvipander’seye,leftinfrared-sensitoright.tivepitPhotographorgan
wmemorksbrane,likeasusppinholeendedcamera.freelysoasRadiationtomiennimizeteringheatthroughlosstothetheopeningsurrounding(left)hitsatissue.Toheat-sensitivensuree
alargeenoughenergyinflux,theaperturehastobequitelarge(∼1mm),comparabletothe
distanceaperture-membrane.Theimageofapoint-likesourcethusformsadisc-shapedimageon
brane.memthe
49
InfraredSystemofSnakes
sensitivecells,freenerveendings(FNEs)thatareorganizedwithindensearchitecturescalledthe
terminalnervemasses(TNMs)[6,121].TheTNMsrepresenttheheatsensitivesensoryareasthat
aredistributedthroughoutthemembrane;cf.Fig.5.2B.Thereareabout40×40sensoryareason
themembrane[198],andthefieldofviewisabout100◦wide,whichimplies◦thatinputtotheorgan
couldberepresentedinthebrainwitharesolutionofapproximately2.5.Sincetheradiationflux
enteringtheorganmustbelargeenoughtoquicklydetectmovingprey,theapertureoftheorgan
iswide,approximately1mm,andcomparabletotheorgandepth.Thusincomingradiationfroma
pointsourcedoesnotstrikeapoint-likeregiononthemembrane,asinanidealpinholecamera,
butrathermuchlargerdisc-shapedregion;cf.Fig.5.1.Itisnonethelesspossibletodetermine
thedirectionoftheincomingradiationiftheboundaryofthedisc-shapedregionremainsnarrow
enough.Thisapproachbreaksdownifmultipleornon-pointlikeheatsourcesarepresent.Thenthe
resultingheatdistributiononthemembranewillbeheavilyblurred(seeFig.5.10),precludinga
directevaluationoftheinput.
TheIR-detectionsystemofsnakespresentsaparadox.Theopticsofthepitorganaspinhole
cameraensuresthatimagesonthemembranewillbedistendedandblurry.Yettheinformationfrom
theIRsystem,combinedwithinputfromthe“normal”visualsystem,allowsformationofarather
preciseneuronalmap[116]inthebrain’soptictectum[109,210,211];cf.seeFig.5.2C.Thismap
issharpenoughtoserveasatopologicalrepresentationoftheoutsideworldinwhichneighboring
neuronsrepresentneighboringregionsoftheoutsideworld.Evenamultimodalinteractionbetween
theinfraredmapandthevisualmaphasbeenfound[109,210,211].Experimentalstudieshave
shownthatasnake’sorientationtoapointsourceofheatvariesbutthatanaccuracyof5◦is
neverthelessattainable[211],whichapproximatesourroughestimateabove.Wearethusfaced
withaparadox,inthattheopticalqualityofapitorganislowbuttheneuronalperformanceis
high.Howdoesthathappen?
TheparadoxinvolvestheopticalqualityofIR“vision”.Wethereforeexploretheoptical
qualityofthepitorganandaddressthethermodynamicphenomenainvolved.Thisenablesus
tocalculatetheheatdistributiononthemembraneforagivenheatdistributioninspace.Ina
secondstep,weaddressthequestionofhowthemappedheatdistributionisencodedneuronally.
Basedonthemembraneimageandtheknowledgeofsensoryencoding,wecontinuewiththe
thirdstepofsensoryprocessing,thatistoneuronallydecodetheoriginalspatialheatdistribution
fromthemeasurementsonthemembrane.Wedeveloptworeconstructionmodelswhichareable
toexplainhowtheneuronalsystemcancompensatetheoversizedpinhole.Sinceasnakehas
limitedcomputationalresources(all“calculations”mustberealizableinneuronal“hardware”)any
reconstructionmodelmustbesimple.Accordingly,ourmodelsuseonlyonecomputationalstep
(theyarenon-iterative)toestimatetheinputimageonlyfromthemeasuredresponseonthepit
membrane.BothresembleaWienerfilterandareakintobutdifferentfromsomeofthealgorithms
usedinimagereconstruction[229].Ithastobeconstantlyborninmindthattheopticaldevice
discussedherehasverylimitedimagingcapabilitiesincomparisontohuman-deviseddetection
systems.Nonetheless,aswewillshow,itisstillpossibletoreconstructthespatialheatdistribution
inabiologicallyplausibleway.Inaddition,wequantifytheperformanceofourmodelsandgive
reasonablelimitationsforlocalizingwarmobjectsbymeansofthissensorysystem.Beyondvarious
specificissuesofthepitsthatareconsidered,weextendthecurrentunderstandingofthissenseby
meansofenlighteningthequestionofwhattheappropriatestimulustotheIR-detetctionsystemof
snakesis.Finally,wepresentanoutlookanddiscussstillopenquestionsthatshouldbeanswered.
50
InfraredSystemofSnakes
Figure5.2:Schematicdrawingoftheessentialpartsofthesnake’sIR-detectionsystem;adaptedfrom
NewmanandHartline[211].(A)depictsapitorganthatisilluminatedbytwoinfraredsources.The
illuminatedareas1and2onthepitmembraneheatup.(B)Sensoryresponsesaregeneratedwithin
thefreenerveendingsofthebranchingtrigeminalnerveaxonsthatformtheterminalnervemass
(TNM).Thespikesaretransmittedalongthetrigeminalnervetothenucleusofthelateraldescending
trigeminaltract(LTTD).(C)Withinthebrain,theinfraredinformationis,i.a.,representedinthe
optictectumbymeansofaneuronalmap.(D)Withintheoptictectumseveralcomputationalsteps
betweenthevisualmapandtheinfraredmapcanbefound.Theinformationofthesetwosensory
systemsfusesintoacomprehensiveperceptoftheenvironment.Foradetailedreviewofthesensory
pathwaywerefertoelsewhere[162,163,182,191,197,198,199,209,211,210,246,255,256,266].
5.2Warmingup:Signalmapping
Inthissection,wederiveequationsofhowtomapthestimulus,thespatialheatdistributionofthe
externalInitially,wweorldclarifyontowhicthehthesensorycellsappropriatewithinstimtheuluspitformemthebraneinfraredofthedetectionsnakeassystemdepictedisandinFig.whatits5.2.
characteristicsare.Insodoing,wederiveanappropriatemathematicaldescriptionofthesignal
cess.promapping
vistadynamicthermoA5.2.1Tostartwiththelatterissue,wefollowthewidelyacceptedopinionthatthepitmembranereceives
thermalradiationfromtheexternalworldandhenceheatsuporcoolsdown,viz.,itisawarm
receptororgan[48,49,168].Ontheotherhand,themembranedoesnotrespondtospecificwave
lengthsliketheretina[93,196].Wehencehavetoanalyzethemechanismsofhowthemembrane
heatsupandcoolsdown.Wederiveasimplifiedmodelofthecomplexheatingandwarmingprocess
toestimatetheimportanceofthedifferentmechanismsinvolved.
Attheoutset,weconceptuallydividetheheattransportphenomenaintotwocategories,the
heattransportwithinthepitmembraneandtheheattransferthroughthemembrane-airsurfaces.
Thatis,weconsiderheatconductionthroughtheair,energygainsandlossesthroughradiation,
51
InfraredSystemofSnakes
thebloodtempperfusion,eratureofheatasmallconductionpitmemwithinbranetheslicetissue,arescandhematicconvallyection.depictedAllintheseFig.pro5.3.cessesaffecting
Figure5.3:Sliceofthecenterofthepitmembrane.Wecanseethedifferentheatflowsthat
gomemthroughbrane-airthesurfacesdifferenStma=surfaces,lp2.Thei.e,edgethemelengthmlpofbrane-memthebranequadraticsurfacessliceisS25mm×=10l−p6dmpandandthethe
thicknessofthemembranedpisassumedto15×10−6m.
Thestartingpointofourstudyisthebasicheatequationthatdescribestheheatdevelopment
withinthetissueofthepitmembrane.Itreads
ρpCp∂T(x,t)=kp2T(x,t)+Qi.(5.1)
t∂iInthissection,wedenotequantitiesassociatedwiththepitswithasubscriptp,forinstance,thepit
membranedensityρp(≈1080kgm−3).TheotherquantitiesareCp(≈3850Jkg−1K−1),theheat
capacitanceandkp(≈0.49Wm−1K−1),thethermalconductivity.Bothrefertothecharacteristics
ofthetissue.Asreasonablevaluesfortheparametersofthepitsarenotavailable,wehavetaken,for
thesakeofconsistency,valuesforneuronaltissue,i.e.,thebrain,fromFialaetal[72].Thefirstterm
intheright-handsideofequation(5.1),kp2Tp(x,t)describesheatconductionthroughthetissue.
Thefundamentalsolution,viz.,incaseQi=0isanexponentialdiffusionT(x,t)∼exp[−x2/ct]of
theheatprofilewithc=const..Theeffectofthistermisthatdifferencesintheheatdistribution
getsmearedout[70].Wewillcomebacktothistermlateronand,insodoing,wespecifyand
discussthedifferentsourcesandsinksofenergyQi.
Thesecondtransportphenomenonwithinthetissueistheheattransportthroughblood.Many
experimentalfindings[6,92,121,205]havereportedadensecapillarysystemwithinthepit
membrane.Amemiyaetal.[5]evensuggestedthecapillariesfunctionasa“heatexchanger”;see
Fig.5.7.Pennes[222]firstformulatedandsubsequentinvestigatorsvalidated[50,72,207,286]the
so-calledbio-heatequationdescribingtheheattransportthroughthebloodQbforpartsofthe
52
InfraredSystemofSnakes
bodysuchasanarm,
Qb=ρbVbCb(1−κ)(Ta−T)=ωCb(1−κ)(Ta−T).(5.2)
HereQbistherateofheattransferperunitvolumeofthetissue.Theperfusionrateperunit
volumeofthetissueisdenotedasVb,ρbisthedensityofblood,Cb(≈4000Jkg−1K−1)isthe
specificheatoftheblood,κisafactorthataccountsforincompletethermalequilibriumbetween
bloodandtissue.Typically,itissetκ=0.ThetemperatureofthearterialbloodisdenotedasTa,
andTdescribesthelocaltissuetemperature.Despitethelargenumberofmitochondriaaround
theTNMs,wedonotassumeintheaboveequation(5.2)thatthepitmembraneitselfproduces
essentialthermalenergyaccordingtobiochemicalprocesses.
Next,weturntothephenomenathatappearatthemembrane–airinterfaces;cf.SmainFig.5.3.
Suspendedwithinthecavityofthepitorganthepitmembranedividesthepitorganintoaninner
andanouterchamber[9,198,211];seeFig.5.2.Inotherwords,themembraneissurrounded
byair.Sincewedealwithverysmalltemperaturesandtemperaturedifferences,aircannotbe
treatedasanidealinsulator.Theheattransportthroughtheaircanbedividedintoconvection
andconduction.Thepitmembraneisalmostperfectlyprotectedfromforcedconvectionbecauseof
itslocationwithinacavity.FreeconvectionwithinanenclosureonlyappearsiftheGrashofnumber
Gr:=gβL3(T−T∞)/ν21000[64].Hereg=9.81ms−2,βisthecoefficientofthermalexpansion,
Lisacharacteristiclength,Tdescribesthetemperature,T∞thesteady-statetemperature,andν
denotesthekinematicviscosity.TheGrashofnumberdescribestherelationbetweentemperature
inducedliftingforceandviscosity.BecauseofthesmalldimensionsofthepitorganGr1000we
canneglectfreeconvectionasBakkenandKrochmal[9]havealreadypointedout.Withthehelp
ofFourier’slawweapproximatetheamountofheatenergylostbyconductionthroughairQaby
Qa=kaT−Tp+kaT−Tp=kadi+do(T−Tp).(5.3)
didodido
Heredianddodenotetheeffectivedistancesfromthepitmembranetothewalloftheinner(di)and
outercavity(do),respectively.Thethermalconductivityofairisgivenbyka=0.026Wm−1K−1,
andTpdenotesthetemperatureofthepitcavitywalls.
Theremainingtransportprocess,namelytheradiationprocess,isthemostimportantonesince
itupmapstheallmembrane,informationandofthistheistheexternalfocuswoforldthisontothesection.memThebrane.secondIntheproblemfirst–place,howradiationinformationheats
istransmittedandextractedbythepitorgans–willbediscussedlater.Wedescribetheenergy
transportduetoradiationprocessesQrby
Qr=−2σT4+µpσTe4+µoσTo4+σTp4(5.4)
bothsidesfrontsidebackside
energylossesexternalworldwarmobjectcavitywall
whereTdescribesthelocaltemperatureofthepitmembrane,asalwaysinthissection.The
meantemperatureofthesnake,viz.itscavitywalls,isdescribedbyTp,andTereferstothe
meantemperatureoftheexternalworldprojectedontothefrontsideofthepitmembrane.In
thefollowing,weassumeTe=Tpbecausethesnakeisacold-bloodedanimal.Thedimensionless
theparametersareaswithµpandmeanµotempdenoteeraturetheTpratioandtheunderwarmwhichatargetsmallobjectsurfacewithoftempthepiteraturememTo;braneobviously“sees”
µp+µo=1.Wecalculatedµo:=A/(2πd2)withA,thevisiblesurfaceofthewarmobject,and
thedistancedbetweenmembraneandobject.Wenotethattheinformationoftheexternalworld
iscontainedintheexactvaluesofµpandµoforeachsensoryareaonthemembranebutforthe
53
InfraredSystemofSnakes
heatingprocess;itsufficestoassume,forinstance,µo≈0.025(d=0.5m)or0.1(d=0.25m).
Furthermore,weassumeallsurfacestobeblackbodyradiatorwiththeconstantσdenotingthe
Stefan-Boltzmannfactor(σ=5.67×10−8Wm−2K−4)emittingtheirenergyintoahalfspace.We
cansummarizethetermsandrewrite(5.4),
Qr=−2σ(T4−Tp4)+σµo(To4−Tp4)
Qr≈−8σTp3(T−Tp)+4σµoTp3(To−Tp).(5.5)
Itcanclearlybeseenin(5.5)thattheenergyinputandtheenergyoutputdependonheatcontrasts,
namely,theexternalandtheinternalheatcontrast.
Thepit-heatequationanditssimplification.Wecannowcombinethedifferentsourcesand
sinksofheatenergywehavederivedaboveintoapitheatequationthatreads
T∂ρpCp∂t=kp2T+Qb+Qa+Qr.(5.6)
Givenallparameters,theaboveequationcanbesolvednumerically.Next,however,wefollowan
analyticapproachtoderivethecharacteristicsofanddependenciesupontheheatstimulus.
Aswehaveseenpreviously,alloftheQitermsdependontemperaturedifferencesinvolvingT.
Wecanreasonablysimplifythesituationbyassumingallpartsofthesnakeandthebackground
oftheexternalworld,exceptthepitmembraneandthetargetobject,tohavethesamemean
temperatureTp.Thisreduces(5.6)toanequationwithonlytwovariables,namely,theheat
contrastintheexternalworldξext:=To−Tpandtheinternalheatcontrastξint:=T−Tp
onthepitmembrane.Furthermore,wecanidentify∂ξint/∂t=∂T/∂t.Inaddition,weleave
thedifferentialdescription(5.6)andconsiderasmallpieceofthepitmembranelocatedinthe
centerofthemembranewithasquaredbaseoflp×lpandathicknessdp;cf.Fig.5.3.Wetake
lp=25µmanddp=15µm.Thelengthlpcorrespondsapproximatelytothesizeofasingle
independentreceptorarea(terminalnervemass).Figure5.3depictstheheatflowsthroughdifferent
surfacesofthemembranecuboid.Themembrane–airsurfacesareSma=lp2=625×10−12m2,the
membrane–membranesurfacesareSmm=lpdp=375×10−12m2,andthevolumeofthecuboidis
Vp=lp2dp=0.01×10−12m3.Thesimplifiedpit-heatequation(5.6)foroursmallcuboidnowreads
τp∂∂ξtint=−Qoutξint+Qinξext,(5.7)
whereτp=ρpCpVp(≈40×10−9JK−1).WecanestimatethevaluesofthetermsQoutandQinby
consideringthecorrespondingmembranecuboid;cf.Fig.5.3.
3di+do4k
Qout=Smaσ8Tp+Smakadido+Smmrˆp
≈8×10−9WK−1≈15×10−9WK−10−10×10−6WK−1
+ωCbVp25×10−9WK−1
≈0.4×10−9WK−1
1Qin=Smaroσ4Tp3100Qout(5.8)
≈0.1−0.4×10−9WK−1
54
(5.8)
InfraredSystemofSnakes
Thestaticpit-heatequationand“optical”contrast.Wenowset∂ξint/∂t=0in(5.7)and
computesimilarlyto[9]themaximalvalueoftheinternalheatcontrastξmaxandthesteady-state
relationbetweeninternalandexternalheatcontrast
ξmax=Qinξout1ξextandξmax1.(5.9)
Qout100ξext100
Wecanseeclearlyfrom(5.9)thatthequalityofthe“opticalsystem”pitorganispoor,i.e.,it
representsexternalcontrastsroughly100timeslessstronglythantheyareintheexternalworld.
Inotherwords,todetectobjectswithabout1Ktemperaturecontrast,snakeshavetomeasure
temperaturecontrastonthepitmembraneintheorderof10mK.Thispurelytheoreticallyderived
resultcorrespondsto,andexplains,manyexperimentalandbehavioralfindings[9,33,63,211].
Figure5.4:Thesteady-stateinternalheatcontrastξmax.Abovewehavestudiedthesteady-stateof
theinternalheatcontrastξmaxofasmallsliceinthecenterofthepitmembrane(seeFig.5.3)by
meansofvarying(A)theexternalheatcontrastξext,(B)thedistancebetweenmembraneanda
warmobjectdMO,(C)thebloodperfusionrateofthemembraneωb,and(D)thedistancebetween
theinnercavitywallandthebacksideofthepitmembranedICM.Exceptifstatedotherwise,
parametersareQin=0.4×10−9WK−1,Qout=50×10−9WK−1,andξext=5K.
discussTogettheatermsdeeperin(5.8).understandingInitiallyof,thetheimpheatortanceconductionofthetermdifferen∼tgradeffects,Tiswsetrnextonglyanalyzesimplifiedand
to∼conduction1/rˆp.Ifwithinweusethethistissuewouldsimplificationthenandclearlytakerˆdominateptoballetheotherradiusterms.oftheThispitapproacmemh,branehowevheater,
istoosimple.Foranumericalcalculationrˆpreferstothedistancelpbetweenthecenterofthe
cuboidandtheneighboringmembranecuboids.Inrealsituations,theneighboringcuboidswill
butshowalmaostsimilarzero.tempNeveratuertheless,reasinthetheobservcaseedthereone.isanHenceexternalthetempheateraturesource,thedifferencesradiatedwillheatnotbeenergyξint
willgradienheattsupatthethememedgesofbranethenotilluminatedhomogeneously;memcf.braneFig.area5.10.thatThewillwbearmingerasedproquiccessklygbyeneratestheheatheat
conductionterm.Inabsenceofsteeptemperaturegradientsonthepitmembrane,i.e.,between
tworeceptorareasinthecenteroftheilluminatedmembranearea,wecanneglectthemembrane
term.conductionheatdepTheendsseoncondthemaanatomjorytermofoftheQpitoutinorgan.(5.8)Thedescribgreateresheatthedistanconductioncebetweenthroughthetheinnerair.caThivitsywtermall
55
InfraredSystemofSnakes
andthebacksideofthepitmembranethesmallerthistermandthebettertheheatcontraston
themembrane[9];seeFig.5.4D.
ThenextimportanttermofQoutin(5.8)isduetoradiationlosses.Amemiyaandothers
[4,38],forinstance,havereportedaspecialsurfacestructureofthepits.Itseemstobeplausible
thattransmittanceandreflectioncouldbeminimizedthroughsuchasurfacestructure.However,
conservabsorbance.ationSiofnceenergythey,aredescribassumededbtoybKirecnearlyhhoff1’s[9la]w,wesetequalsboththethermalthermalradiationradiationemittanceemittanceand
andtheabsorbanceto1.Accordingly,thepossiblevariationofthesetwoparametersisverysmall.
Moreover,experimentalquantificationismissing.
brainThetissuelast[72],termthisofQtermoutisdescribquiteessmalltheheatandlossesunimportanthroughtasbloweodhavfloew.IfdepictedwetakineFig.the5.4valuesC.Bufort
asreportedinmanyexperiments,themembraneshowsagreatamountofcapillariesanditisof
verysmallscalecomparedtothetissueusedforderivingthebio-heatequation[222].Moreover
andunfortunatelywithoutgivingnumbersontheinterestingtimescale,Gorisandcolleagues
[92,121,205]havereportedthatthebloodflowwithinthecapillariesfollowsthetemperature
development.Theyalsospeculatedaboutare-coolingmechanismoftheTNMbymeansofthe
bloodflow,whichevenseemstobecontrolledlocallybytheTNM.Accordingly,weeitherhaveto
modifytheequationbymeansofconsideringω=ω(T)(seeFig.5.4C)orwehavetomodelthe
effectofbloodflowotherthanPenne’sbio-heatequationbecausethedimensionsofourproblem
aremoretwo-dimensionalinsteadofthree-dimensional.
Ontheotherside,Qinandthusξmaxisinfluencedbytheexternaltemperaturecontrastξextas
wehavedepictedinFig.5.4A.Thegeometryofthesetting,inmathematicaltermsµo,isinfluenced
bythesizeofthevisiblesurfaceAandthedistancedMObetweenthemembraneandthewarm
objectofinterest.ThedistancedMOlimitsthemaximalheatcontrastξmaxandthusthedetection
rangeaswehaveillustratedinFig.5.4B.
Thetime-developmentofthepit-heatequation.Anotherimportantaspectof(5.7)isthe
timedevelopmentoftheheatcontrastandthusatheoreticalapproachtotheresponsetimeofthe
infraredsystem.Wecanrewrite(5.7)tothedifferentialequation
∂∂ξtint=−τ1ξint+Qˆin,(5.10)
wherewehavesubstitutedτ:=τp/QoutandQˆin:=Qinξext/τp.Thegeneralsolutionoftheabove
equationisgivenbyasuperpositionofthehomogeneoussolution(Qˆin=0)andaparticular
solution.Combined,thesolutionreads
ξint(t)=ξmax1−exp−t,(5.11)
τwithξmax:=τQˆin=(Qin/Qout)ξext–theresultforthesteady-statecondition(5.9).Underthe
assumptionsandsimplificationswehavemadeabove,wereceiveatime-dependendequationthat
allowsustostudythequestionofwhatinfluencesthewarmingprocessofthepitmembrane(5.11).
InFig.5.5wehaveillustratedthedependenciesofthewarmingprocessupontheinfluencing
variables.Asaspin-offwefindphysicalevidenceforexperimentalresults[33]thatsupposea
tempneuronaleraturerespcononsetrastbeloonwthe100msmemusingbrane,mKisnottempconstaneraturetduringresolution.thatpTheeriodsignal,oftimehowbutever,i.e.,increasing.the
Inthissection,wehaveforthefirsttimederivedanequationincludingallaspectsofenergyin-
andoutputofthepitmembranewarmingprocess(5.6).Bysimplifyingthisequationto(5.7)we
havequantifiedboththedependenciesoftheheatcontrast(5.9)andthetime-dependentwarming
process(5.11).Thusweareabletoidentifytheimportanteffectsandshowwhichdataarecurrently
missing.Fromtheexperimentalpointofview,wehavearrivednotonlyatatheoreticalvalidation
56
InfraredSystemofSnakes
Figure5.5:Transientwarmingprocessofthepitmembrane.Abovewehavestudiedthewarming
ofasmallsliceinthecenterofthepitmembrane(seeFig.5.3)bymeansofvarying(A)the
externalheatcontrastξext,(B)theamountofenergylostbythemembraneQout,(C)theamount
ofenergyreceivedbythemembraneQin,and(D)parametersofthepitmembraneτpincludingfor
instancethemembranethickness.Thebasicgraph(red)correspondsinallpanelstotheparameters
Qin=0.4×10−9WK−1,Qout=50×10−9WK−1,τp=40×10−9JK−1,andξext=5K.
and,moreimportantly,ataclarificationofthepossiblesensoryperformancesbuthavealsoprovided
mathematicaltoolsforfurtherinvestigations.
Inthenextsection,weturntotheinformationcontentthatistransmittedtothepitorgan.
Thatis,wewillclarifytheradiationprocessinmoredetailbymeansofdisentanglingµoforevery
sensoryareaonthewholemembrane.
heattheMapping5.2.2Haandvingthattheclarifiedaccuracythatonlyofthetheheatradiationconoftrasttheplaysexternalacritiwcalarmrole,objewctsehacanvetocauseaanalyzereasonablecarefullystimwhiculush
inphysicaltegratedspdescriptionectralfitsradiancebestu(toλ,Tm)ap=πtheI(λ,T)externalderivheatedbyconMaxtrastPlantocktheinthe1901pit[226]membrane.The
u(λ,T)=2hc52πhc1Js−1m−3,(5.12)
λexpλkT−1
describestheamountofenergyablackbodywithtemperatureTemitsintothehalfspaceper
time,perwavelengthλ,andperunitsurface.AswecanseeinFig.5.6,almostall(99,5%)ofthe
tenthradiatedoftheenergytypicalfrompitablacorganskbapodyerturewith[47T,=137303].KWeisecanmittedthereforebelowignore100µmeffectswhichthatisareaboutdueoneto
thethatwais,vethecharactermappingofoflighthetandspatialinsteadheatusedistribgeometricution.Asopticsatoside-remark,describethephatmosphericysicsoftranoursmittanceproblem,
canbeignoredoverdistancesofmeters[8];sowecanneglectthiseffect,too.
Next,wecondensethephysicsofgeometricalopticsintoaformalismthatmapstheemitted
signalsfromtheexternalenvironment,forinstance,apreyontothepitmembraneofasnake.In
theprevioussubsection,wesawin(5.7)thattheexternalheatcontrastξextisanappropriate
57
InfraredSystemofSnakes
Figure5.6:Thespectralradianceu(λ,T)(left)andthenormalizedintegralofu(λ,T)(right)asa
functionofthewavelengthλattemperaturesofT=303K.Weseethatthewavecharacteroflight
canbeignoredandwecanusegeometricoptics.
descriptionforthestimulusbecauseQin–aconstant,mainlygeometricfactor–timesξextisthe
receivedenergyonthemembrane.WenowleavethesimplifiedQin,whichisadescriptionforone
areaofthepitmembranebymeansofµoandµpin(5.5),andderiveadescriptionforallsensory
areasionthepitmembraneaccountingforanarbitraryspatiallydistributedexternalheatcontrast.
WethereforediscretizetheexternalspaceinfrontofthepitintoplainsurfacesjofequalspaceSext
withtheheatcontrastξext;jfromnowondenotedbyξj.Usingthegeneralformalismderivedin
Chap.2,thereceivedenergyriduetoradiationofasensoryareaionthepitmembranenowreads
ri=hijξj.(5.13)
Themappingismathematicallymodeledbymeansofthetransferfunctionhijcomponent-by-
componentwherethecomponentsofthetransferfunctionare
jSmaSextσTp31/d2ijcos(ϕ),ifiisvisiblefromj
hi=0,otherwise.(5.14)
Toaccountfortheconservationofenergy,theemittedenergyscalesas1/d2ijcos(ϕ)wheredij
denotesthedistancebetweenthesurfacesjintheexternalworldandionthemembrane.Werefer
toϕastheanglebetweenthenormalvectorsofthesurfacesiandj,andthereforejthecosineterm
compensatesfordifferentorientationsofthetwosurfacesiandj.Thetensorhiisonlydetermined
bythegeometryofthepitorgansandthemeantemperature.Wecanassumebothtobeconstant.
So,thetensorprojectseveryheatdistributionoftheexternalworldinthesamewayontothepit
brane.memToexplicitlycalculatetheresultinginternalheatcontrastonthemembrane,onehastosolve
(5.7)foreachsensoryareaigivenri,oronecouldlinearize(5.7)andjustmultiplyribythe
appropriatefactorderivedfrom(5.7).Aswewanttostudyinformationcontents,wewillusethe
pureheatintensitydistributionriassensoryinputinthefollowing;cf.Chap.1.
Forrealisticallymodelingthemappingprocessoftheheatintensitydistributionontothesurface
ofthesensorycellsiofthemembrane,wealsohavetoconsideruncertaintiesandnoise.Two
sourcesofnoisecanbeidentified.Noiseofthesignalandmeasurementerrorsofthedetector,
respectively.Wemodelbothbymeansofaddingstochasticvariablesαjandχitotheinputand
thedetectorresponse
ri=hij(ξj+αj)+χi.(5.15)
Thenoiseofthesignalcanbecausedbymeansofdisturbancesofotherobjectssuchasgrassand
fastmovements.Sinceeverydetectionprocessisblurredbynoise,itisreasonabletoassumethe
samehere,too;cf.Chap.2andChap.3.
58
InfraredSystemofSnakes
Insummary,wehaveidentifiedtherelevantphysics,i.e.,geometricaloptics,andderiveda
modelthatdescribestheheatintensitydistributionacrossthepitmembrane.Inthenextsection
weturntothedetectionprocessitself.
Figure5.7:Capillarynetworkandterminalnervemass(TNM)ofthepitmembrane;adapted
from[5,6].Theupperpanels(A−B)depictscanningelectronmicrographsshowingthecapillary
vlosingasculaturetheirofsheaththepit(smallmemarrobrane.w)andThe(Dlo)wtheerfreepanelsnervilleustrateendings(C)(nf)theofmayeterminallinatednervnerveefibmassersarra(mfy).
Acollectionoffreenerveendingsformsareceptorunitandrespondstotemperaturestimulias
modeledinSec.5.3.Experimentaldataonthenervous-controlledcapillarynetworkanditseffect
onthewarmingprocessaremissing.
dingencorySenso5.3Inthissectionwewanttoslightlytouchuponthequestionastohowthestimulusisencoded
neuronallytoextractimportantideasfortheoverallfunctioningandperformanceoftheinfrared
es.snakofsystemdetectingFreenerveendings–amulti-purpusedetector.Weknowthatthelocationwherethe
stimwithinulustheisfreedetectednerveistheendingspit(FNE)memofbrane.theMoreterminalinnervparticularemass,the(TNM)sensoryofpitrespmemonsesbarerane.Ingeneratedother
words,thefreenerveendingsencodethetemperatureoftheirsurrounding(staticresponse)and
theyfoundinrespondothertoanimalstempersucaturehascmice,hangesvampire(dynamicbats,resporonscatse)[[3324,,117118].,F243ree,244nerv].eInendingsthesearesettings,also
vFNEsalidit[y.41]First,areknothewnstaticasresp(mammalian)onsecurvcoldesorw(neuronalarmreceptors.activityvs.Twotempfindingserature;seemcf.tobeinlaofyunivFig.ersal5.8)
ofandthe[24,235thermo].Second,receptorsinttohedifferenanimalstlevaboelsveof–constanalthoughtnottempaseraturepronouncedareassimilar;inthecomparesnake’s[117pit]
59
InfraredSystemofSnakes
membrane–thereisalsoadynamicresponsetotime-dependentthermalstimuli;cf.Fig.5.8and
e.g.[243,244].InpassingwenotethatFNEsseemtobeusedalsoinspecificwatersnakesfor
detectingwatermovements[43].
Figure5.8:Averageimpulsefrequencyofapopulation(N=9)ofwarmfibersfromthetrigeminal
branchwhenapplyingthermalstimulitotheskinofaBoaConstrictor;adaptedfrom[117]and
[25].Duringthisexperiment,Henseletal.haveincreasedthetemperature(redcoloredareaand
righty-axis)step-wise(ΔT=5◦C),startingatT=20◦CandendingatT=40◦C.Aswecan
see,themaximaofthedynamicresponseaswellasthesteady-statevaluesofthestaticresponse
(seeinlaygraphC)followabell-shapedcurve.Theinlaygraphshowsthesteady-stateresponse
curvestostatictemperaturestimulusof(A)acat(N.lingualis,bursting),(B)adogfish,(C)aBoa
Constrictor,and(D)acat(N.infraorbitalis,non-bursting).Notonlytheresultingdetectoractivity
isqualitativelycomparablealsotheunderlyingdetectionprincipleseemstobethesame:freenerve
endings.
Amodelforthermoreception.Thestaticresponsecurvesofmammalianthermoreceptors,the
so-called“coldfibers”,andtheunderlyingbiologicaldetectionmechanism–freenerveendings–seem
tobethesameoratleastcomparabletowhatwefindinthepitmembrane;cf.Fig.5.8.Wehave
thereforeadaptedacommonionicmodelformammaliancoldfibers,theHuber-Braunmodel[26,27,
28,124],tostudyFNEresponses.Notsurprisingly,wecanreproducetheexperimentallyobserved
staticresponses,abell-shapedfrequencyvs.temperaturecurvature.Evenmoreinterestinglyand
forthefirsttimewehavebeenabletoreproducethedynamicresponses;seeFig.5.9.Toour
knowledge,nootherthermo-receptormodelhasbeenabletodoso.
TheHuber-BraunmodelforthermoreceptorsisbasedonasimplifiedversionoftheHodgkin-
Huxleymodel.Themodelconsistsoftwosetsofionicconductancesthatoperateattwodifferent
voltagelevelsandtimescales.Thetemperaturedependenceismodeledbytwoscalingfactorsfor
therateconstantsρ(T)andthemaximumoftheconductancesΦ(T).Themainobservableisthe
membranepotentialVthatiscontrolledbyfivecurrentsaccordingtothemembraneequation
VdCMdt=−INa−IK−Isd−Isr−Il,(5.16)
whereCMisthemembranecapacitance.Theioniccurrentsoftherighthandsidecanbegrouped
intothreecategories.ThefirsttwofastcurrentsINa/Kgeneratetheactionpotentials.Thenext
60
InfraredSystemofSnakes
twocurrentsIsd/srareslow,subthresholdde-orrepolarizing,andtheirinterplaycausesmembrane
potentialoscillations.Finally,thelastcurrentIlisthepassiveleakcurrent.Allcurrentsare
modeledinthefollowingway
Ii=ρgiai(V−Vi),(5.17)
withρatemperaturescalingfactor,githemaximumconductance,aitheactivationvariable,and
Vithereversalpotential.Theactivationvariableaiitselfcanbedescribedbyadifferentialequation.
ForadetaileddescriptionoftheHuber-Braunmodelanditsnumericalparameterswereferto
elsewhere[26,27,28,124].Inthelightofourfocus,theHuber-Braunmodelcanreproducethe
neuronalresponsestostatictemperaturelevels,e.g.,thebell-shapedactivityvs.temperature
characteristic;seeFig.5.8.Yetthemodelcannotreproducetheobservedneuronalresponsesto
eratures.ptemhangingcTheHuber-Braunmodelanditsextension.Wemodifytherepolarizingsubthresholdcurrent
Isr=ρgsrasrf(V,T)in(5.17)(correspondstoequation(5)in[27])byintroducingatemperature-
gradient-dependentactivationvariableasrthrough
asr(t>t0)=asr(t)−ΔasrifgradT|t=t0=0.(5.18)
Thismeansthatifatemperaturestepoccurs,theactivationvariableasrisreducedbyΔasr=0.5.
Itismathematicallyunimportantwhetherwemodifyasrortheconductancevariablegsr.The
resultisthatIsrisreducedshortlyafterachangeintemperature,andtheneuronfiresaburst
ofspikes.Thismodificationcorrespondsinprincipletoatransientionicgatingmechanismdue
to,e.g.,thermo-sensitiveTRP(transientresponseprotein)channels[23,176,189,212,275]which
resultsinanadditionalcurrentoccurringattemperaturechanges.Bothatemperature-depending
ionicgatingmechanismbymeansofTRPchannels[97]andanadditionaltransientcurrentsensible
fortemperaturechanges[219]havebeenfoundinthesnakerecently.
TheresultofournumericalexperimentisdepictedinFig.5.9andagreeswellwithbiological
findingsofthesnake’sneuronalresponses;cf.Fig.5.8.Thatis,theextendedHuber-Braunmodel
cannowreproducetheneuronalresponsestotemperaturechanges.Moreprecisely,ourfindings
hinttowardsthreeimportantissues.
•First,evenifthemechanismofthedynamicanswerismuchmorecomplex–aswebelieve–
itcanbeunderstoodatfirstsightbymeansofionicmodelsincludinganadditionalthermo-
gradient-sensitivecurrentorchannel.
•Second,wewanttohighlightthatatemperaturechangemodeledinthewaydescribedabove
synchronizesthedifferentFNEs.WepredictthatsynchronizingeffectsofFNEresponses
willenhancereliability,justasinmanyotherneuronalsensorysystems[80,81].Aprecise
quantification,however,willneedamoredetailedanalysis.
•Third,wecanseeaprominentdifferenceinthequalityofthestaticandthedynamicanswer.
Thetemporalprecisionandthesuddenchangeinfrequency(≈80Hz)atatemperaturestep,i.e.,
thedynamicanswer,ascomparedtotheincrease(≈5HzforΔT=5◦C)inthecorresponding
staticresponseisveryhigh.Inotherwords,wecandefinetherelativequalityΩofhowprecisely
theFNEssystemcanencodeatemperaturechangebymeansofachangeofthecorresponding
neuronalactivityfromATatTtoAT+ΔTatT+ΔT,
Ω=AT+ΔT−AT.
AT
(5.19)
61
InfraredSystemofSnakes
Figure5.9:SimulationofFNEresponses,thesensoryencodingstep.Theneuronalactivityof
uncoupledneurons(N=26),theaveraged(timeintervalsare200ms)firingrate,isplottedon
the(left)y-axis.TheneuronsaremodeledaccordingtothemodifiedHuber–Braunmodelof
thermo-sensitivecoldreceptors[27,28,124]asdescribedinthemaintext.Theneuronssequentially
start50msaftereachother.Duringournumericalexperiment,wehaveincreasedthetemperature
(redcoloredareaandrighty-axis)step-wise(ΔT=5◦C),startingatT=5◦Candendingat
T=30◦C.Aswecansee,themaximaofthedynamicresponseaswellasthesteady-statevaluesof
thestaticresponsefollowabell-shapedcurvecomparabletothebiologicaldata;cf.Fig.5.8.The
x-axisispresentedpiece-wise(e.g.fromt=t1tillt=t1+5s)aroundthetemperaturestepsbecause
thetemperaturestepssynchronizetheoutputoftheFNEsandpreventareasonableaveraging.We
concludethatthemodified(seemaintext)Huber–Braunmodelcanreproducethesensoryencoding
(FNEsresponses)oftemperaturedynamics.ThatisTRPchannelsresultinginasub-threshold
currentsthataresensitivetothermalgradientscancausethedynamicresponse.
Wecannowseethatthedynamicanswerencodestemperaturechanges:temporallypreciselyandvery
reliably;Ωdyn≈80Hz/20Hz=4.Incontrast,thestaticresponsetoabsolutetemperatureislesspre-
cise,relativelyweak,andevenambiguousontheleveloffrequency;Ωsta≈5Hz/20Hz=1/4=Ωdyn/16.
Thisbecomesevenmoreevidentifweadditionallyconsidertheamountoftimeinwhichthechange
ofactivityΔAoccurs.
encodedbAccordinglyythe,wstaticecanrespsponseeculatebutthatinsteadtheusessnakedochangesesnotofusethethetempabsoluteeratureteconmptrasteratureencocondedtrastby
thedynamicneuronalresponse.Thepresentparagraphshaveshownthatthedynamicanswerand
yettempboralelievced.hangesNext,ofwethewillheatexploitcontrasttheonresultstheofmemthebranepresenpttossiblywoplasections.yamoreThatimpis,weortancomtrolbineethanour
knowledgeonthephysicaltransmissionprocessandthefindingsofthesensorydetectionprocessto
deriveaplausiblemodelfortheneuronalreconstructionmechanism.
62
InfraredSystemofSnakes
dingdecoNeuronal5.4Inthissection,westartwiththesimplestmodelwecanimaginethatisabletoreconstructthe
heatcontrastdistributionξintheexternalworld.Thisminimalmodelaccountsforthelimitation
ofaneuronalsystembymeansofrestrictedcomputationalpower.Weextendtheminimalmodel
bymeansofincorporatingpropertiesofthesensoryencodingprocesssoastoarriveataplausible
andmeaningfuldescriptionoftheinfrared-visionsystemofsnakes.Inaddition,wetestourmodels
invariousconditionswithregardtoexternalheatcontrastandomnipresentnoise.Finally,we
showthatandpredicthowfeatureextractionispossibleusingboththestatic(minimalmodel)and
dynamic(extendedmodel)responsesoftheinfrareddetectorsofthepitmembrane.
delmoMinimal5.4.1Wehavetothinkofsimplemodelsbecausethesnakehaslimitedcomputationalresourcesinview
ofproptheosefactamothatdelallthatiscalculationsnon-iterativmustebandeusesrealizableonlyinoneneuronalcomputationalhardwarestepandthatfast.canbWeethdoneerebfyorea
feedforwardnetwork.Thereconstructedexternalheatcontrastdistributionξˆjatlocationjcan
thereforebewrittenasalinearcombinationofthemeasuredintensitiesri,
ξˆj=ljiri.(5.20)
Theoptimalreconstruction.Tofindtheoptimalcoefficientsofthereconstructiontensorlji
wefollowtheframeworkofoptimalmapformationpresentedinChap.2anddefinetheexpectation
valueEofthequadraticerrorofourestimateξˆias
E=(ξˆi−ξi)(ξˆi−ξi).(5.21)
Intheaboveequationweinserttheexpressionξˆi=lijrj=lijhjk(ξk+αk)+χjfrom(5.15)and
(5.20).Furthermore,wecomputetheexpectationvalueoverallpossibleinputαianddetector
χinoises.Thenoise-freepartoftheinputξiisdeterministic.However,wewanttooptimally
reconstructnotonlyonespecificsituationbutthetypicalIRenvironment.Inotherwords,
biologicallyrelevantsignalsbelongtoaclassofsignalsthatwedenoteby“typical”.Consequently
aspecificsensorysignalisaconcreterealizationofatypicalbiologicallyrelevantsignal;henceit
isastochasticquantity.Wethereforehavetodealwiththeexpectationvalueofthesignal;fora
detaileddiscussionofwhywehavetotaketheexpectationvaluewerefertoSec.2.2.2.
ibeInzerothbeecausefollowingthenoisecalculationstermsofareδEindep/δhjenden=0,tofweeachassumeotherallandappofearingtheinputcrossstrengths.correlationsTheto
correlationtermscanbeapproximatedtofirstorder
χiχj=σχ2δij,αiαj=σα2δij,andξiξj=σξ2δij(5.22)
whereδijdenotestheKroneckerdelta.Allthreeequalitiesdefineautocorrelations.
Thefirsttwotermsdenotethenoisetermsontheinputandthereceptormanifold.Herewerefer
tonoiseasaspatiallyindependentadditiverandomvariablewithzeromeanandstandarddeviation
σχ/αthatmodelsthefollowingpoints.Evenifthedetectionmechanismonthepitmembraneis
thefast,asignalsignalwillinbetheavexterneragedaltowtheorldrelevcanbanetfaster.signalSothatwecancanbedefineanreconstructed.integrationThislowindow-passwwithinfilter
effectisinherenttoanysensorysystem,nomatteriftechnicalorbiological.Theinputnoiseαi
accountsforthiseffectandthereforeblursthetruesignalξi.Strictlyspeaking,thenoisewould
thenbespatiallycorrelatedduetoacommonmovementoftherealIRobject.However,wewill
63
InfraredSystemofSnakes
Theignoresecondthisknonoisewledgetermanddescribthusesdealtheinthedetectorfollonoisewingonwiththeawpitmemorst-casebranescenariothatofmayknowingoriginatenothing.from
physiologicalprocessesintheheat-sensitivecells.
Thethirdtermin(5.22)describestheautocorrelationoftheinputthatwehavemotivated
abovdeviatione.Inσofgeneral,theitinputcanξb=e(.w.r.,ξitten,..as.);ξiseeξj=Chap.(µξ22.+Wσeξ2)δwillijonwithlymuseeanoneµξtermandconthetainingstandardthe
iξtnowypicaldependsinformationontheabpoutersptheectivIReofsurrounding,whethertheviz.,relevanteitherthestimulusmeanortheinformationstandardisdefineddeviationas.theIt
te.g.,ypicalpreyh.eatThecontrastformerorcaseastherefersvtoariationtheofmeanatµξypicalandheattheconlattertrastcasetorelatingthetostandanobardjectofdeviationinterest,σξ.
Forwithoutthesaklosseofofuniformitgeneralityybinyσourξandskipmathematicalforthetimedescription,beingwecathenabodenotevethequestionimpofortantwhatquanistitthey
relevantstimulustothepitorgan.Indoingso,weassumethatthecorrelationbetweeninputsat
differentlocationinspacei=jvanishes.Thatis,aworst-caseansatzagainbecauseknowingthe
valueoftheinputatoneparticularpointnowdoesnottellussomethingaboutitsvalueatother
pfromositions.thesameThisobjectassumptionortheofsamecoursewillsurrounding.notholdHeatinrealittranspybortecausephenomenaneighboringpeliminateointstesteepndthoebate
howgradienever,ts,lessandthantheptempossibleeraturemakleveselofneighreconstructingboringthepoininputtswillonlymorethereforebedifficult“similar”.andsufficesAssuming,fora
h.approacminimalHavingclarifiedthemappingoftheinputontothepitmembrane(5.15)andhowwecandescribe
ljtheiasnoisedefinedandininput(5.20).cThatharacteristicsis,weha(5.22),vetowenextminimizeprotheceederrorfinding(5.21)thewithoptimrespalecttoreconstructionthecoefficientensorts
ofthereconstructiontensor,i.e.,δE/δhji=0.Definingthedimensionlessparameterσ:=σχ/σξ
andτ:=σα/σξwecancomputethecomponentsofthereconstructiontensorljisoastofind
likhjvhkv(1+τ2)+σ2δjk=hji.(5.23)
toNextrealwequantakeatities.briefThelookatparameterthetwτomodenotesdeltheparametersinverseτandσ,signal-to-noisetheirmeaning,ratioofandthetheirinputwhicrelationh
ingeneralwillbesmall.Thesecondparameterσrelatesnoiseonthepitmembranetothemean
vki.e.,inputitintenincreasessityinitstheeigenvexternalalues,wtheorld.valueSinceofσσdirectlyinfluencesmothedifiesthereconstructiondiagonaloftensorthematrixsignifican(hjtlyh.v),
64
InfraredSystemofSnakes
Figure5.10:Minimal-modelreconstructionperformance;figuresareadaptedfrom[252].We
discretizeAlbrechtD¨urer’s1502paintingofahare(left)intoapixelizedversionwithan8-bitgray
scaleintendedtorepresentthehare’ssurfacetemperature.Inacomputermodel,thisIR-source
imagewaspresentedtoanidealizedsnakepitorgan.Thedashedcircleindicatestheorgansfieldof
view(100◦).MappingtheIRsourcebymeansofthetransferfunctionhgivenby(5.14)ontothe
pitmembraneresultsinaheavilyblurredmembraneheatimage(middle).Inthenextstep,the
modelreconstructstheIRinput,i.e.,theharefromtheIRimageonthemembraneofthemodelpit
organ.Therightpanelsshowthereconstructedimagebymeansofthereconstructionmatrixlof
(5.23)calculatedfordifferentnoiselevelsofthemembranesensorcells(0.25%,1%,2%,and5%of
themeanvalueofthemembraneintensities).Forfurthermodelparameterssee[252].Despitethe
pooropticalqualityofthepitorgan(centerimage),theoriginalinformationcanstillbeobtained
neuronallyfromthemembraneimage.
Thereconstructionperformanceoftheminimalmodel.Toillustratetheabilitiesofour
model,wepresentanexplicitexampleofhowpreciselythesnakecanreconstructitsIRsurroundings.
Weusethefamous1502drawingofAlbrechtD¨urerofahareconvertedinto8-bitgraylevelsata
resolutionof32×32;cf.leftpartofFig.5.10.Inthenextstep,wecalculatetheheatintensity
distributionrionthemembraneusingthetransferfunctionhkvgivenby(5.14).Wehavechosen
conservativebutrealisticvaluesthatcombinearelativelywideorganaperture(radius0.4mm)and
amoderatenumberofheatreceptors[252].Thecalculatedmembraneheatdistributionisshownat
thecenterofFig.5.10.
Theresultsofapplyingthereconstructionalgorithmtothemembraneimageareshownin
Fig.5.10.Thequalityimprovementisspectacular,providedthereisnottoomuchdetectornoise.
Themostimportantresultisthattheinformationneededtoreconstructtheheatpanoramaisstill
presentalthoughinvisibletooureyeinthemembraneimage.Thepresentmodelhasafairlyhigh
inputnoisetolerance.Forinputnoiselevelsupto50%,thehareisrecognizable.Sensitivityto
measurementerrorsislarger.Inourcalculations,onepixelofthereconstructedimagecorresponds
toabout3◦.Fordetectornoiselevelsuptoabout1%ofthemembraneheatintensity,agood
reconstructionispossible,meaningthattheedgeoftheharemaybedeterminedwithaboutone
pixelaccuracy.Atdetectornoiselevelsbeyondabout1%,theimageisnotsoeasilyrecognizable.
Choosingahigherσcancompensatethemembranenoiseuntilacertainnoise-levelrevealingat
leastthepresenceofanobject;cf.Fig.5.10.Thedrawbackofchoosinghighσvaluesisthatdetails,
viz.,variationsoftheheatimageare“treated”asnoiseandwillbesuppressed.Inaddition,in
thecaseofasimpleandclearinput–say,asmallwarmanimalinfrontofacoldbackground–
reconstructionmaystillbepossibleathigherdetectornoiselevelsaswewillseelater.
Predictedneuronalcharacteristics.Ourmodelisnotonlyabletopredictthesensoryperfor-
manceofthesnake’sIRsystembutcanalsoshedlightontotheneuronaldecodingmechanismused
tosharpenthemembraneimage.Theneuronalimplementationofthemodelisfairlystraightfor-
65
InfraredSystemofSnakes
Figure5.11:Predictedreceptivefields.Asexplainedinthemaintext,thestrengthoftheindividual
connectionscorrespondstotheentrylji.Theneuronsinthetopologicalmapreceiveexcitatory
(positive,red)aswellasinhibitory(negative,blue)inputfromthemembranereceptors.Thespatial
distributionoftheseexcitatoryandinhibitoryregionsonthemembraneiscalledthereceptivefield
ofacorrespondingmapneuron.Aswecanseeinpanel(A−E)thereceptivefieldforasingle
mapneuronconsistsofaexcitatoryringlocatedattheedgesoftheheatprofilesurroundedbyan
inhibitoryring.Thisconstructionconstitutesanedgedetector.Wecanshrinktheradiusrofthe
organaperture(B:0.4mm,C:0.2mm,D:0.1mm,andE:0.05mm)untilwereceiveapin-hole-like
geometrythatiscomparabletothesetupinopticalsystemsemployingalens.Inbothsensory
systems,theIRsenseandthevisualsystem,wefindthesamemechanismsthatfollowthesame
mathematicalprinciplesofoptimality.Reconstructionparameterσ=3,distancebetweenaperture
andobjectis1m,anddistancebetweenapertureandmembraneis0.8mm.
ward.Thereconstructiontensorljicorrespondstoanetworkofsynapticconnectionsbetweenthe
membraneIRdetectorsandneuronsbuildingamapin,forinstance,theoptictectum[109,210,211].
Thestrengthoftheindividualconnectionswouldjustbethevalueofthecorrespondingentrylji,
basedonaratecoding.Theneuronsinthetopologicalmapreceiveexcitatory(positive)aswell
asinhibitory(negative)inputfromthemembranereceptors.Thespatialdistributionofthese
excitatoryandinhibitoryregionsonthemembraneiscalledthereceptivefieldofacorresponding
mapneuron;cf.Chap.1.ItcanbecalculatedbyourmodelasillustratedinFig.5.11.Ring-like
structuresarisethat“detect”theedgesofprojectedimagescorrespondingtothepositionthatthe
mapneuronencodes.Asimilarsystem(calledlateralinhibition)isfoundinmammalianneurons
receivinginputfromtheretina;cf.Chap.2.Althoughtheexcitation-inhibitionpatternthere
isdisc-shaped(“small”aperture)ratherthanring-like(largeaperture)theexcitation-inhibition
patternnaturallyfollowsthesameprincipleofmathematicaloptimalityasthereceptivefieldwe
havederivedhere.
66
InfraredSystemofSnakes
Insummary,thesnakedoesneedaccuratedetectorstogetareasonablereconstructionquality.
Indeed,theexperimentallydeterminedmembraneprecisionishigh.Thepitmembraneresponds
totemperaturechangesdowntoabout1mK[33,63].Aswediscussedinpart5.3,theresponse
ofthedetectors(FNEs)totemperaturechangesismuchmoreprominentandthusreliablethan
tostatictemperaturevalues.Wenowincludethisfindingandadapttheminimalmodelderived
aboveaccordinglybymeansofincorporatingthedynamicdetectorresponses.
delmoextendedThe5.4.2HereweextendtheminimalmodelwithrespecttothemajorfindingofSec.5.3concerningthe
neuronalcharacteristicsofsensoryencoding.Themajorresultisthattemperaturechangescould
beencodedneuronallymorereliably,i.e.,withamuchhighertemporalprecision,andmoreclearly
thanthestaticmembranetemperature(5.19).Wenextdiscussthisfindinginthelightofneuronal
decodingandadaptourminimalmodelderivedaboveaccordingly.
Detectorcharacteristicsinthelightofneuronaldecoding.Theminimalmodelclearly
informationdemonstratescanthatbethedecodedinformationneuronally.transmittedThetominimalthemopitdel,memhowbraneever,isfacesstillatherebigdraandwbacthatk.theIt
needshighlyaccuratesteady-statetemperaturemeasurementsonthemembraneandtheliterature
wlaceksreferanetoxpthelanationfindingsofofwhereSec.the5.2.1necessarywhereweprecisioncouldseecouldthatoriginatethetempfrom.eratureTogivconesometrastnisuminbtheers,
rangeofmKwithinthefirst100msofastimulusonset.Inaddition,themeasurementerrorsneed
is,tobtheelessFNEsthanhav10%etotodetectreconstructabsoluteaheattemppanoramaeraturecasonwetrastshaveofseabenout1previously;mKupcf.toF20ig.mK5.10.withThatat
leastsampling0.1inmKtervalsaccuracyto.encoGivdeenthearelevnoiselessanttempdetectioneratureprovcess,alues.theLookingdetectorsatthewouldneuronalthenrespneedonses200
(seefoundFig.inthe5.8)toeascendingncodenerthevesranofgetheofpitΔξme=m20braneK;ev(Fig.enif5.8)weweconsiderfindathefrequencytemporalrangeofpatternsayof20Hzthe
thecorrespsensoryondingsystemspikeandtrainsalmaosthighunbelievaccuracyableinincaserepresenofatingnoisythestimdetectionulusproseemscess.veryWecthereforehallengingnoforw
focusontheotherneuronalresponse,thedynamicansweroftheTNMstotemperaturechanges.
Theextensionoftheminimalmodel.Wecanthinkoftwodifferentoriginsoftemperature
changesofthemembrane.Thefirstpossibilityisthesudden(dis)appearanceofaheat-emitting
toobjecmotvemencausingts.Wthearmriseoborjectsfallinofthethememexternalbraneworldtemptranslatingerature.Therelativsecondelytoandthepimosttmemnaturalbraneisleaddue
toobjectemptmovorallyes,ctemphangingeraturetempchangeseraturesappateartheatthesensoryedgescofellstheontheresultingpitmemtempbrane.eratureIfpsucrofilehaonwarmthe
membrane.Inencodingtemperaturechanges,theFNEsdonothingbuteffectivelyfilteringthe
edgesWeofcanthepitcalculatememanbraneinedge-filteredtensityversiondistributionfi(seeri.Fig.5.12)ofthemembraneintensitydistribution
rjbymeansoflinearfilteralgorithmssuchastheLaplacefiltermatrixtij[91],
fi=tijrj=tij(hjvξv)=hˆivξv.(5.24)
vˆTheheatabconovetrastξvequationontoandefinesedge-filteredsimilarlytointensit(5.14)yaprofiletransferfionfuncthetionmemhibrane.thatnoSincewmapsthetheexedge-filteredternal
imageapproximatesthetemporalderivativeoftheintensityprofileonthemembrane,wehave
essenextendedtialpropourmoertiesdelofofthethesnakdetectionemecinfraredhanism.senseinUsingathesimplesamebutmethoreasonabledologywaasythatdescribnoedwiinncludes(5.20)
67
InfraredSystemofSnakes
–(5.23),wecancalculatecorrespondinglyto(5.23)anoptimalreconstructiontensorlˆijusingthe
newtransferfunctionhˆivof(5.24)
lˆikhˆjvhˆkv(1+τ2)+σ2δjk=hˆji.(5.25)
Arrivingattheaboveequationwehaveimplementedaveryplausiblesecondstepofsensory
processing(seeChap.1),i.e.,wehaveincludedthepitmembranedetectorcharacteristics.Even
thisseemstobeasmallstep–aswehaveleftoutthederivationstepsanalogueto(5.20)–(5.23)
fortheminimalmodel–itresultsinabigadvantageaswewillseeinaminute.
Figure5.12:TheLaplacefilter.Asexplainedinthemaintext,weusetheLaplacianfilterthat
can“detect”theedgesofthemembraneintensitydistributiontoextendtheminimalmodel.The
Laplacianfiltermechanismworksinthefollowingway:(B)Thefilterkernelsequentiallytakesevery
pDepoinetofndingtheontheoriginalLaplaceimage,filterscaleskernelit,atandhand,subtractsitcantheaffectweighthetedareavofalue3×of3,itsasinneightheboringfigurep5oin×5ts.,
ormorepixelscenteredaroundtheevaluatedpoint.(C)Tousethismethodinthecontextofthe
minimalmodelwehavetoiwritethefilterkernelaccordinglytotheindexingioftheoriginalimage
inimagetothe(A)linesontoofatheLapmatrixtlacianjforfilteredeachimageimage(pBoin)bt.yThemeansofLaplaceonematrixmatrixtmjnowultiplication.mapstheoriginal
advAdvanantagestagesthatofthearisefromextendedusingmothedel.dynamicTheinsteadextendedofmothedelstaticfeaturesresponse.thefolloFirst,wingthechangeconceptualin
neuronalactivityΔA≈80Hzofthedynamicresponsecausedbyanincreasingtemperature
contrastξisabout16dyntimeslargerthanΔAsta≈5Hzofthestaticresponsecausedbyanincreasing
tempemployerature16timescontrasmoret;cf.Secsampling.5.3.poinDuetstotothisdecofact,detheusingsignal.theThatdynamicmeans,responsetheallodetewsctiontheacsnakcureacyto
es.vimproΩ,theSecond,relativnoteconlyhangetheofabsolutefrequencychangecomparedoftoneuronaltheactivitactualyΔAfrequencyoftheoffreethenervedetector,endingsisbutincreasedalso
(5.19).Third,Thatacis,hangingthequalitIRyenandvironmenreliabilitt–ymoofvingtheobneurjectsonal–rdoespesonsenothevenindermorebutthanactuallyimproenableves.the
detectionprocess.IncaseofusingtheabsolutetemperaturecontrastξoftheIRenvironment
(minimalmodel),ξhastobestatic.Thatis,incaseoftheminimalmodel,movementsaretreated
asnoise.Incontrast,theextendedmodelisbasedonthedynamicresponse,i.e.,detectorresponses
tothechangessignaltoofthethemoIRendel.Thisvironment,decreasesdξ/dt.thTheus,noisemovlevelemenintsthearenotsignalrelatedmappingtoprnoiseocessbut.Ofconstitutecourse,
thereisalsoatemporal“integrationwindow”duringwhichasignalhastobeconstant.Sincethe
68
InfraredSystemofSnakes
dynamicresponse,i.e.,thesudden(about100ms)increaseinfrequencyoftheneuronal-detector
activity,istemporallyquitepreciseandfast,theintegrationwindowisnarrow,andthusthe
precise.aredetectorsFourth,weremembertwofindings.First,inSec.5.2.1wehavederivedthatthetemperature
contrastξonthemembranehasnotreacheditssteady-statevaluewithintherelevantreaction
timeofthesnake.Thesignalξcanthusneverbeconstantandself-averaging,respectively.Second,
thereconstructionmechanismneedsveryaccuratesensorydata.Asignalthatisconstantduringa
certainperiodoftimecanbedeterminedmorepreciselybymeansofmakingmoremeasurements
andaveragingtheresults.Inthecaseoftheminimalmodel,ξisnotconstantandthuscannotbe
improvedbystatisticalmethods.Onthecontrary,thetimederivativeofthetemperaturecontrast
dξ/dtisalmoststaticwithintherelevanttimeframe.Astaticphysicalsignalprovidesaveraging
and√thusreducingadditivedetectornoise.Wecanatleastimprovethemeasurementbyafactor
∼twithmoremeasurements.Thatis,theaccuracyofhowwecandetectandencodethesignal
ishigherincaseoftheextendedmodel.
Insummary,wehavederivedanextensionoftheminimalmodelthatconnectssmoothlytoa
plausibleunderstandingofthedynamicresponsepatternfoundintheFNEs;seeFig.5.8.Toclarify
theoverallsensoryperformanceoftheextendedmodel,wenexthavetoquantifyitsdependency
uponnoise.SincetheunderlyingmathematicsoftheextendedisakintoaLaplacian-filteredversion
ofthemembraneimageonecannotexpectagoodnoiserobustnessbeforehand.Thisfactcould
cancelouttheaboveadvantages.Inthefollowingsection,wethereforecomparetheabilityofthe
twomodelsathandtoreconstructtheIRenvironmentandtoextractrelevantfeaturessuchasthe
objectpositioninvarioussettings.
extractioneatureF5.4.3AfterhavingseenabovethattheIR-sensorysystemofsnakescanreconstructtheheatenvironment,
wobejecnotswandaddresswhichthequanquestiontitiesofinfluencehowthepreciselysensorysnakpes,thaterformance.isourtwomodels,canlocalizewarm
Thenumerical“experiment”setup.Inthefollowingnumericalexperimentswepresenta
warmobjectwithinanon-uniformandnon-zerotemperaturebackgroundtoanartificialpitorgan.
WeassumeanIRenvironmentwith25◦Cmeantemperature.AswederivedinSec.5.2.1and5.3,
theexternalheatcontrastξextandthecorrespondingtimederivativedξext/dt,respectively,are
theappropriatestimuliforthepitorgan.WeaddaGaussian-distributedtemperaturebackground
contrastwithΓ◦CmeanandΓ◦Cstandarddeviationtostudytheeffectsofexternalnoiseand
signalcontrast.ThewarmobjectshowsaGaussian-shapedtemperatureprofilewithmaximalheat
contrastξmax(inrespectto25◦C)andstandarddeviation6.25cmthatislocatedatx=(x,y)
withx,y∈[−1m,1m],1minfrontofthepitorgan;cf.Fig.5.13.Foreveryrunwerandomly
positiontheobjectcenteratx,addanoisybackground,mapthewholeinfraredsceneontothepit
membraneandreconstructtheenvironment.Weusesimpleshapesbecausewenowcanextractthe
estimatedpositionxˆ=(ˆx,yˆ)easilybymeansoftakingtheaverageofthe50%warmestlocations
ofthereconstructedenvironmentandcomparetheestimatedpositionxˆtotherealpositionx;
cf.Fig.5.14.Duringthenumericalexperimentswemonitorthedependenceofthelocalization
performanceuponthebackgroundcharacteristics,namely,Γandthedetectornoiseχ;seeFig.5.15.
WehavetonotethatthefocusofthisworkisontheIRsensorysystemandnotonthemost
cognitivealgorithmforobjectrecognitiongivensensorydata,i.e.,apost-processingstep.Wehave
alreadyshown[251]howtoestimateboth,positionandshapeofobjects,givensensorydataofa
naturalsensorysystem.So,ourfindingspresentabottom-lineperformanceofthesnake’ssensory
abilitiesthatcanbebetterthanderivedhere–intheaspectofobjectrecognition.
69
InfraredSystemofSnakes
Figure5.13:Setupforlocalizationperformanceexperiments.(A)Wesimplifythewarmobjecttobe
atwo-dimensionalGaussian-shapedprofilewithstandarddeviation6.25cm,maximalamplitude
ofξmax=5◦C,andpositionx,andaddaGaussianrandombackgroundtemperaturecontrast
distributionwithmeanandstandarddeviationofΓ◦C.(B)ThisparameterizedIRenvironment
ismapped(minimalmodel:hikorextendedmodel:hˆik)ontothepitmembrane,anddetector
noiseχiisadded;cf.(5.15).Nowitdependsonwhichmoidelweconsider.Eithertheintensity
distributionriandthecorrespondingreconstructiontensorlk(minimalmodel)ortheedgesofthe
intensitydistributionfiandlˆik(extendedmodel)areusedtoreconstructtheIRenvironment.In
thelattercasetheedge-filteredmembraneimagefiapproximatesthetime-derivativefi≈dri/dt
duetoobjectmovements;seeSec.5.4.2.(C)Inthefinalstep,weextracttheestimatedpositionxˆ
ofthewarmobjectfromthereconstructedIRimageandcompareittotherealpositionxduringn
independentruns:seeFig.5.14.Toquantifyhowtheparametersaffectthesensoryperformance,
wepresenttheroot-mean-squareerrorbetweenx=(x,y)andxˆ=(ˆx,yˆ)inFig.5.15.
Sensoryperformance“experiments”.Withinthesetupoutlinedinthelastparagraph,we
varythesignal-to-nosie(SNR)ratiooftheexternalheatcontrastξbymeansofincreasingΓgiven
aconstantξmax=5K.Inaddition,westudyhowthedetectornoiseχionthepitmembraneas
givenby(5.15)affectssensoryperformance.Theresults(Fig.5.14)illustratehowpreciselythe
positioncanbeestimatedfromtheIRenvironment.Incaseofmoderatenoiseboththeminimal
model(leftpanelsinFig.5.14)andtheextendedmodel(rightpanelsinFig.5.14)canreconstruct
thepositionofthewarmobjectquiteprecisely.
Inthissection,thestandarddeviationofthemembranenoiseσχisrelatedtothemaximal
valueofeithertheintensitydistribution(minimalmodel:σχ∼rmax)ortheedgesoftheintensity
distribution(extendedmodel:σχ∼fmax)onthemembrane.Thepreviousdefinitionofmembrane
ofthemeanvalueofthemembraneintensities:σχ∼1/niri;seeFig.5.10.Sinceahigher
noisepresentsthestandarddeviationσχoftheadditiveGaussiannnoisevariableχiinpercentages
temperaturebackground–viz.,increasingΓ–alsoincreasesthemeanvalueofintensitiesrion
membrane,thetwodifferentkindsofnoisewouldnotbeindependentincaseoftheprevious
definition.Soweredefinethenoiseleveltobenowrelatedtothemaximalvalueofthemembrane
intensityriortheedgesofthemembraneintensityfi.Thisre-definitionalmostcompletelydecouples
.ΓandχFurthermore,weseeinFig.5.14thatinbothmodelsthecapabilitytolocalizeawarmobjectis
stillavailablebeyondthemembranenoiselevelof1%butdecreasesfast.Independentlyofeach
otherboththeexternaltemperaturebackgroundΓandthemembranedetectornoiselevelsχ,play
animportantroleinpredictingthesensoryperformanceandinevaluatingexperimentalresults.
Surprisingly,theestimatedpositionsxˆdonotonlybecomemorevaguebutalsoclusteredaround
themiddleofthepitapertureduetoincreasingnoiselevels.Thelattereffectismoreprominentin
70
InfraredSystemofSnakes
respcaseofonsesthetomitempnimaleraturemodel;ccf.hangesleftinpanelsprincipleinFig.access5.14.sensoryThedataextendedwithmohigherdelemploaccuracyyingasthecompareddynamic
totheminimalmodel.Onthecontrary,sinceLaplacianfilterstypicallyenhanceimagenoise[91]
wasecouldcomparedexpetoctthethatmintheimalextendedmodel.moIfdelthisthatisisthebasedcase,onthealessLaplaciannoisefiltertoleranistlessdecodingtoleranttoalgorithmnoise
wouldcancelouttheadvantagesgainedthroughamoreaccuratesensoryencodingmechanism.At
firstglance,theextendedmodel,however,isasrobusttonoiseastheminimalmodel;seeFig.5.14.
Figure5.14:Localizationperformanceexperiments.Thetwocollectionsofninepanelsillustratethe
effectsofthetwoparametersχi,themembranenoise(x-axis)andΓ,parameterizingtheexternal
IRenvironment(y-axis)forboththeminimalmodel(left)andtheextendedmodel(right).Ineach
ofthe18panelswehavedepictedrealpositionsx(panelx-axis)versusestimatedpositionsxˆ(panel
y-axis)of2500trailscorrespondingtothesettingdescribedinFig.5.13.Weseeclearlythatboth
modelscanreconstructthepositionoftheinfraredobjectverypreciselyincaseofmoderatenoise
levels.Inbothmodels,localizingawarmobjectispossiblebeyondthemembranenoiselevelχ
of1%butdecreasesfast.Furthermore,thelevelofcontrastΓwithintheexternaltemperature
backgroundplaysanimportantroleinpredictingthesensoryperformanceasthelocalizationgets
worsewithincreasingΓ(seey-axis).Interestingly,notonlydoestheestimationaccuracydecrease
butforhighΓtheestimatedpositionstendtobemoreclusteredaroundthecenteraswell.Above
all,theextendedmodelrobustnesstonoiseseemsatleasttobecomparablewiththeminimalmodel
–itisevenbetterintheparameterrangedepictedabove;butseealsoFig.5.15.
Tomathematicallyquantifytherobustnesswithrespecttobothtemperaturebackgroundand
detectornoise–theperformanceofthetwomodels–wedefinetheroot-mean-squareerrorErms
betweenrealxandestimatedcoordinatexˆ
Erms=(x−xˆ)2/n(5.26)
nthroughrobustnessnasindepcomparedendenttoruns.theInextendedFig.5.15model.weseeThethatnoisethetoleranceminimalofthmoedelextendedshowsmoadel,higherhowevnoiseer,
isgoodenoughtoallow,incombinationwiththeadvantagesofthesensoryencodingstep,i.e.,the
higherqualityoftheneuronalresponse(5.19),theconclusionthattheextendedmodelisthemodel
e.oichcofConsequently,weagainposethequestionofwhatisthestimulustotheinfraredsystemsincewe
havederivedthesuperiorityoftheextendedmodelemployingthedynamicresponse.InSec.5.2.1
71
InfraredSystemofSnakes
wehavederivedthatthestimulushastobetheexternalheatcontrastξextaccordingtothe
thermodynamicsofthesysteminsteadoftheexternalabsolutetemperature.Thefindingsof
sensoryencodingonthelevelneuronalcompartmentsinSec.5.3havethenhintedtowardsthetime
derivativeoftheexternalheatcontrastdξext/dttobetheadvantageousandappropriateinputfor
thepitorganinsteadofξextortheabsolutetemperature.Thenumericalperformanceexperiments
havenowunderlinedthisassumption.Althoughintheliteraturetheabsoluteexternaltemperature
isreferredtobeingthestimulustothesensorysystemwecanalsofindexperimentalstatements
thatsnakesrespondbesttomovinginsteadofstationarywarmobjects[94].Aswehavediscussed
above,thelatterstatementscorrespondtodξext/dtandhenceprovideanexperimentalevidence
forourtheoreticallyderivedfinding.
Figure5.15:Quantificationofthemodels’localizationperformance.TheplotsshowErmsforthe
andminimalχ(y(left)-axis)andusingtheanextendedoptimalmoσ.delFor(righeacht)insetdofepnoiseendenceupparametersontheΓnoiseandχ,wparametersehaveΓv(xaried-axis)σ
The(2500minimalrunsformoeacdehlcomemploysbination)alargertofindparametertheoptimalspaceinσthatwhichlominimizescalizationtheofloawarmcalizationobjecterrorisErquitems.
wgooelldpenoughossible,toviz.,notincancelwhichoutErmstheisadvsmall.antagesTheofnoiseusingthetolerancedynamicoftherespextendedonsesucmohasdelistheathigherleast
qualityofencodingatemperaturestimulus.
Insummary,wehaveshownmathematicallythattheIRsignalsoriginatingfromanexternalheat
distributionandgettingprojectedontothepitmembranestillsufficetoreconstructboth,animage
oftheexternalheatdistributionandthepositionofwarmobjects.Furthermore,wehaveclearly
demonstratedthatincludingcharacteristicsofthedetectionprocessintothereconstructionmodel,
viz.,focussingonthedynamicresponse,enablesareasonableunderstandingoftheremarkable
performanceoftheinfraredsystemofsnakes.
72
InfraredSystemofSnakes
5.5DiscussionandOutlook
Inthischapterwehaveanalyzedtheinfraredsystemofsnakesbystudyingthethreestepsof
sensoryprocessingthatwehadintroducedinSec.1.1.First,usingtheeyeglassesofphysicswe
havelookedatthemembraneheatingprocessandcorrespondingexperimentalfindingsofthepit
organssoastoidentifytherelevantequationsdescribingthefirststepofsensoryprocessing,viz.,
thesignalmapping.Indoingso,wehavequantifiedhowtheheatingprocessdependsonexternal
quantities–suchasdistanceortemperaturecontrastofawarmobject,anatomicaldata,and
time.Second,wehavepresentedanionicmodelthatcombinesrecentanatomicalfindingsand
presentmodelsofheatreceptors.Thatis,bytakingtheeffectsoftransientresponseprotein(TRP)
channelsandtime-dependentsub-thresholdcurrentsintoaccountwewereabletoreproducethe
neuronalresponsesofthefreenerveendings(FNEs)notonlytostatictemperaturesbutalsoto
temperaturechanges.Ourfindingsofsensoryencoding,furthermorehinttowardssynchronization
effectsoftheFNEsthatcouldberesponsiblefortheoutstandingtemperaturesensitivityofthis
sensoryarchitecture.Third,forthelaststepofsensoryprocessing,i.e.,neuronaldecoding,wehave
providedbothaminimalandanextendedmodelthatareabletoreconstructtheexternalheat
distribution,i.e.,theinfraredimageoftheenvironment,givenonlytheheavilyblurredmembrane
image.Acomparisonofbothmodelswiththeirresponsetodetectorandexternalnoisehasrevealed
theabilitiesandlimitationsofthisIRsense.Ourfindingsbothfitwellintotoday’spictureand
extendtheunderstandingoftheinvolvedprocesses.
Moreover,wehaveposedandansweredthequestionofwhatisthetruestimulustothis
sensorysystem.Inthefirstsectionofthischapterwephysicallyderivedthatitisnottheabsolute
temptemperatureerature.butIntheaddition,temptheeraturecequationsontrasthavofetheshownexternalthatwtheorldtempthateratureeffectivconelytrastaffectsonthethememmembranebrane
doesnotreachitssteady-statevalueduringtherelevantresponsetime.Sincethetimederivativeof
thetemperaturecontrastonthemembranestaysalmostconstant,wehavestartedtoconsiderthe
wetimehavederivfounativdeofmoretheevidenceexternaltoheatsupportconthistrasttothesis.betheThestimsupuluseriortoqualittheyIRofthesystem.sensoryInthisrespconsehapterto
temperaturechangesascomparedtostatictemperaturesisoneofthem.Totesttheassertion,we
haveextendedtheminimalansatzmodelingthereconstructionperformance.Insummary,wehave
thenotbtempeenoralabletoderivfindativeorofderivtheeexternalreasonableheatconargumentrasttsistheagainstbutappropriateonlysintimfavulusorforforthethesnakclaimes’thatIR
system.Withinthiswork,notonlyarenewinsightspresentedbutalsonewquestionsforfurther
researchhavearisen.Inthecontextofsignalmapping,experimentallygainedvaluesoftheinvolved
quantitiessuchasthebloodflowkinematicsortheheatconductanceofthemembranetissue
areneeded.Systematicandquantitativemeasurementsofthewarmingprocessdynamicswould
allowforunderstandingthefunctionsoftheindividualphenomenasuchasbloodflowandsurface
structureone-by-oneandininterplaywithoneanother.
dynamicsOnthearelevelneededoftosensoryfurtherencoenlighdingtenmothedelsquestionincorpoforatingwhatexpmakeserimenthetalpitmemdataofbraneTRPsocsensitivhannele.
ApromisingissuecouldbesynchronizationeffectsoftheFNEs.
Inthecontextofreconstructionmodelsweseetwofutureapproaches.Ontheonehand,
theoreticalmodelsshouldbecomemorebiological.Thatis,theyshouldbebasedonneuronaldata
–ifpresent–todescribesensoryperformanceonthelevelofspikingneurons.Theextendedmodel
basedoncharacteristicsofthesensoryencodingprocess,i.e.,thedynamicresponse,providesagood
startingpoint.Ontheotherhand,nobodyhaseveraddressedthequestionofhowtheaccuracyof
theIRsenseandthusthetopographyofthecorrespondingneuronalsynapticconnections,arises.
Thereisaclearlackofexperimentaldataonthisissue.Fromthetheoreticalpointofview,onecan
imagineinstructivelearningasfoundintheauditorypathway,orsomekindofauto-focusalgorithm
73
InfraredSystemofSnakes
thatusesimagestochasticstoimproveitself,tocausethedevelopmentofthissensorysystem.In
thenextchapterwewillstudyamodel,calledintegratedMultimodalTeaching,thatcouldbea
reasonableexplanationofhowtheinfraredmapandthevisualmapintheoptictectumgetaligned.
Thefindingsderivedinthischapterarenotpurelyofbiologicalinterest.Asourmathematical
modelsshowagoodperformancewithoutemployingcomplexcalculations,wehavealreadystarted
tosetupaprojecttoimplementourfindingsintohardware;cf.Fig.5.16.Theoverallgoals
aretoevaluateourmodelsinrealsettings,toisolateindividualaspectsofthemodelsandthe
involvedphysicsthatarenotpossibletostudyinabiologicalcontext,andtodevelopandtest
learningalgorithms.Onthehardwaresidethegoalsaretoclarifythepossibilityofimplementing
bio-inspiredcognitivecircuitsinsilicontoimproveopticalIRinstruments.Firststepsarepromising,
andwearelookingforwardtothisinterdisciplinary,bionicapproach.
Figure5.16:Bionicexperimentsetup.Asdescribedinthemaintext,wearegoingtosetupan
artificialpitorganexperimentincooperationwithErikJungandAlexanderHilgarthfromthe
FraunhoferIZM,Berlin.Theinputprojectormapsthesignalontothedetectorsoftheartificialpit
organ.Werealizeafullyflexiblemotor-controlledgeometry(flexiblepositionofthepitorganwith
respecttotheinput,variableorganaperture,adjustabledistancebetweenapertureanddetector)
toevaluatethepredictionswehavederivedinthiswork.Moreover,wewilldevelopandimplement
“reality-feedbacked”learningalgorithmsandminiaturizethissetupintheprogressoftheproject.
Afinalremarkregardingthesensoryperformanceoftheinfraredsystemofsnakes:Itseemsthat
notonebigdevelopmentbutratherthecomplexinterplayofseveralspecificstepssuchasavery
thinandisolatedmembrane,adensecapillarynetwork,FNEsemployingtemperaturesensibleionic
channelsthatareorganizedtoacompactnetworkcalledterminalnervemass,neuronaldecoding
mechanisms,etc.haveledtooneofthemostspecializedandremarkablesensorysystemsfoundin
nature.Wewereabletoanswersomebutnotallofthequestionsconnectedwiththesesteps.
Inthenextchapter,wefocusonthegeneralquestionofhowaprecisealignmentofdifferent
sensorymapscandevelopintrinsically.Thealignmentofdifferentmapsisfundamentalfor
multimodalintegrationthatisalsofoundinthesnake’soptictectum.There,theinfraredmapand
thevisualmapinteractfaithfully,forinstance,bymeansofperforming“AND”(e.g.visibleand
warm)and“OR”(e.g.visiblebutnotwarm)operations[109,210,211].
74
Millionssawtheapplefall,butNewtonwasthe
onewhoaskedwhy.
hBarucesMannBernard
6.IntegratedMultimodalTeaching
Inthischapterwecomebacktothetopicoflearningandaddressglobalstructuresunderlying
theevolvementofsensorysystemsandtheirpreciseinteraction.Weintroducetheconcept
ofintegratedMultimodalTeaching(iMT)thatisbasedonsupervisedspike-timing-dependent
plasticity(sSTDP)andmultimodalintegration.Weshowthatamultimodalteachercanensure
properintrinsicmapformationandalignment.Thedominanceofvisioninguidingmapalignment
isdemonstratedtobeanaturalconsequenceoftheexceptionalprecisionofthevisualsystem.
ductionIntro6.1Ourabilitytoactinadynamicandcomplexenvironmentreliesfirstandforemostontheprecise
interactionofthedifferentsensorymodalitiesthatformacongruentperceptionoftheexternal
world.Thispreciseinteractionofthedifferentmodalitiesisnotpresentatbirthbuthastobe
learnedduringthefirstperiodofsensoryexperienceinlife[271,280].
Figure6.1:Thebasicideaofintegratedmultimodalteaching(iMT).Allmodalitiesareintegrated
aintoteacanhinginsignaltegratedm(blackultimoarrodalws)mapthat(graleadsyarrotoanws).Theadaptationactivitofyofthetheunimomultimodaldalmaps.mapWethenexplaininducesthe
detailedmodelsetupemployedinthepresentchapterinSec.6.2.2.
Theprocessofsensorycalibrationiscommonlyseenasvision-guided;cf.King’sreview[148].
Severalremarkableexperimentsshowthatdestructionordisturbanceofthevisualpathwayleads
todisorganizedandabnormalsensorymapsinnon-visualmodalities.Thesefindings,obtained
insupporthamsterthe[200idea],ofcat[vision278,as280teac],clahingwedmofrogdalit[y.44],Howferretever,[150],vision-guibarndeowld[map159],andcalibrationinsnakisesnot[96as],
evidentasitseemsatfirstglance.First,sensorymaps,althoughimprecise,canevolvewithoutany
visualdespiteainput[missing146,158visual,271].system,Exptheerimentsauditoryonandvisuallydeprivsomatosensoryedanimalssystemsandcanhumansdevelophaveshonormally;wnthatsee
[152,231,234]fordetails.Second,behavioral[30,68,77,249,258]andphysiological[202]studies
75
IntegratedMultimodalTeaching
haWveallaceshownandthatStein[vision277,can279]halsoavebepoininflutedencoutedbythatotherreceptivmoedalities.fieldsinThird,thesupanderiormostimpcolliculusortan(SC)tly,
thatalreadyallmoshrinkdalitiesatshotimeswawheretempornoallyvisualcoincidingneuronsdevareelopmenpresentt.inFwhichurthermore,evenvisiontheyhavimproeves.observButed
ifhowvisioncanitshouldimprovacteasitselfteac?Thingogether,modalitthey,abhoowvecanparitadoguidexicalcaexplibrationerimentalwithoutandbeingtheoreticalpresenfintdingsand
questionthecurrentpictureofvisionasthepredominantguidingmodality.Thatis,whatisa
coherentalternativetovisionastheteachingmodality?
moIndalitiesthiscbeginshapter,atwtheetaktimeetofwoappfindingsearanceasofmstartinultimogpdaloinints.tegrationFirst,[the279].formationSecond,ofthedifferenmotor,t
hasi.e.,bameenultimosuggesteddalmaptoinprothevideSCaasteacwhellingasinsignalitsfornon-mammalianauditoryspacehomolog,alignmenthet[optic180];cf.tectumFig.(OT),1.4
forSCconnectivity.Therefore,wesuggestanintegratedMultimodalTeachingconcept(iMT);
seefindingsFig.as6.1.toWhoewexplaindifferentthemwoell-prodalitiesvencanbedominanalignedceofwiththerespvisualecttosystemeachandother.unifyIncondoingtradiso,ctorwey
inontroaduceconceptualaninlevtegratedel.Tmoultimothisdaend,lteacweherfocusandondiscussmapitsinformationtrinsiccwithinharacteristicstheSC/OTandbyimplicationsmeansof
long-termpotentiation(LTP)andlong-termdepression(LTD)ofsynapsesthatcanbedescribedby
movsSTDPe.theirWeleasensoryveoutorgansthe(eyquestiones,ears,ofwhiskdynamicers,etc.)adaptationindepasendenapptlyeariofngtheirinbmanodyy.Theanimalsmecthathanismcan
underlyingthisadaptationisnotknown,butretinotopiccoordinatesystemsseemtoplayakey
horolewev[83er,].mAttemptsultisensorytoandmodelthispredominanissuetlycanmonbeosreviewensoryedlayiners[82are,99m,240utually,241].alignedWithintogaintheaSC/OT,unified
multisensoryrepresentationofsensoryspace[147,261].Thecombinedsensoryinformationcan
Inthenotherbewusedords,toatgeneratethisstagedirectionalallsensoryresponsesmapsinsharetheaSC/OTcommonmotorrepresenmap[tation166,167system.,179,Sim216il,ar260to].
othercalibrationtheoreticalappearingmodelsin[the78,con287],texttheofthesuggesteverydiMTcommonconceptsensoryconcenspace.tratesInonadditionthetoquestionderivingofmapthe
generalconceptforiMT,weapplyourmodeltoconcreteexamples.Usingthismodel,wereanalyze
ofinWdetailallacetheandconcreteStein[277exp,er279ime]nintsoftheKnlightudsenofouretal.iMT[159]concept.andcompareFinally,wthemewithconcludethebyfindingsgiving
experimentallytestablepredictionsofourapproach.
6.2IntegratedMultimodalTeaching(iMT)
Ourconceptofintrinsicsensorymapcalibrationstructurallyconsistsofastackofunimodal
maps,eachconnectedwiththemultimodalmap(cf.Fig.6.1).InouriMTconcept,different
unimounisensorydalmapsmocdalitiesancurrenarefirsttlybemergedfoundintowithinthemtheultimoSC/OT,dalmap.whereSucthishamsituationultimoisdalexpinerimentegrationtallyof
wellunimodaldescribmaps.ed[261Th,us,279].theInmtheultimonextdalstep,maptheinducesinategratedteacmhingultimosignaldalthatmapproguidesjectsthebackcalibrationtothe
does(formationnotoriginatandealignmenfromat)singleprocessmoofdalitythe,fordifferenexample,tunimovisiondal,butmaps.fromtheTherefore,integratedtheteacmultiherminodalput
map.InForgeneral,adetailedmapcalibrationunderstandingcanofbeourdividedmodelintodesign,twopleasedifferentseeSec.classes.6.2.2.Thefirstdescribesthe
situationinwhichsensorymapshavetobelearnedfromscratch.Connectionsbetweensensory
systems(inputmaps)andrepresentingmapscontainnoinformationandarerandomlydistributed.
Thesynapses;inconnectionshamathematicalvetoevtermsolveduringdescribedthebySTDPcalibration.WeprodenotecessbythismecaseansasofLTPformationandofLTDtheofmathpse.
Thesecondclassrequiresacollectionofalreadypre-wiredmaps[271]andcorrespondsto
76
eachingTdalMultimoIntegrated
anstimulusalignmentpositionofismapheres,dforisplacedinstance,induringcomparisongrotowththeorpinositionshiftingoftheexpteacerimenhertsinput.[158,By159].meansTheof
activity-basedsynapticplasticity,thesynapticpatternofthemapismodifiedsoastocompensate
forthemisalignmentbetweenthemapandteacherinput.
TocarefullyanalyzetheconsequencesofapplyinganiMTsignalwehavetounderstandmap
proformationcess(asandshowninalignmenFig.tf6.2)romaproconstitutesceduralapclosedointoffeedbacview.kloToopthisthatend,consistsweofnotethreethatreptheeatingiMT
ansteps.inFirst,tegratedmduringultimomultimodaldalmap.integrSecond,ation,duringdifferentmultimosensorydaltemoachingdalities,theareinoptimallytegratedmultimergedmoindalto
mapalignmeninducest,ofanthediffereninhibitorytteacsensoryhingmaps.signalThird,thatguidesduringtheunimodalcalibrationmapproadaptioncess,,i.e.theformationadaptingandand
adaptedunimodalmapsagainmodifytheintegratedmultimodalteacher.
Thethreephaseswehavedescribedaboveareinherentlyconnectedbymeansofthreemajor
questions:1)HowdounisensorymapsdetermineiMT?2)HowdothecharacteristicsofiMT
together,influencethesemapthreeadaptation?questionsandpro3)videHoawdostructureesiMTforourcalibrateiMTdiffeconceptrentandunimwilloadaldmaps?ditionallyTakguideen
usthroughtheanalysisoftheiMTprocessinthefollowing.
Figure6.2:MainproceduralaspectsofiMT.Combiningdifferentunimodalmapsintoamultimodal
mapleadstocertaincharacteristics.Whatarethesecharacteristicsandhowdotheyforman
instructiveteachingsignalforthemapcalibrationprocessthatcanbedescribedbymeansof
supervisedspiketiming-dependentplasticity(sSTDP)?Toclosethefeedbackloop,wediscusshow
differentunimodalmapsadapttoacommonteachingsignal.
6.2.1HowdounisensorymapsdetermineiMT?
aTosensoryanswermothedalitfirstymasaitsjorneurquestion,onalwerepreseninitiallytationhavonetoaclarifyspatialthemap;essenseetialSec.terms.1.2.WOnerefersuchtoa
maponecaninterpretthefiringrateasthelikelihoodtofindastimulusatthepositionthatthe
neuronsencode.Inthismanner,sensorymapsrepresentpositionestimatorsorpopulationcodes
[60,116,131,227].Dependingonthestatisticsofthefiringprofile,wecanattributeameanµ
correspondingtotheestimatedpositionandastandarddeviationσdescribingtheestimator’s
accuracytoeachsensoryestimator.Theintegrationofdifferentsensorymapscanthereforebeseen
tobeequivalenttoacombinationofpositionestimators[2,67,102,113,126,165,202].
Althoughtheexactneuronalmechanismastohowdifferentmodalitiesareintegratedintoa
multimodalmap[60,61]isstilllargelyterraincognito,thefamilyofBayesiancombinationschemes
mathematicallydescribestheresultofoptimalmultimodalintegration.Forthesakeofsimplicity,
wewillfocusonthemaximumlikelihoodcombinationofonlytwomodalitiesinthefollowingtext,
77
IntegratedMultimodalTeaching
Thenamelyvisual,visionand(V)auditorandymapsauditionare(cA).Theharacterizedcasebofythreetheirmomeans,i.e.dalitiesisthepdescribositionedinestimates,AppendixµVC.2.for
visionandµAaudition.Furthermore,theaccuracyofthepositionestimatesisgivenbyσV(vision)
andσA(audition),andillustratedbyFig.6.3.
forFigureaudition,6.3:Multimovision,dalandinthemtegrationultimooftdalwomapunisensoryassumedasmaps.TheGaussianfiringfunctionsprofileswithofmeanneuronalµA/Vmaps/M
inandtegratedstandardmapisdeviationassumedσtoA/Vb/eMincantegratedbeinbymterpretedeansofasapmositionaximumlikestimatorselihoodofthemethodstimurepresenlus.Theted
byand(6.1)isthandusthe(6.2).most(A)Theprecisemofultimoalldalmaps.map(Bsho)IfwsthethesmalleunisensoryststandardmapsaredeviationshiftedσµMV<=σVµA,<σtheA
multimodalmapµM(redprofile)islocatedmorecloselytothemoreprecisemodality(blueprofile)
.(6.3)(see
ThemultimodalpositionestimateµMasamaximumlikelihoodcombinationfromcoinciding
visualandauditoryinputsisgivenby[135,144]
22σσµM=σA2+VσV2µA+σA2+AσV2µV(6.1)
inwithtegration,thestandardthemultimodeviatidalonσMestimatorofthemfeaturesultimothedalpsmallestositionpesossibletimate.Asstandardaconsequencedeviationofσgivoptimalen
MybσσVAσM=σA2+σV2.(6.2)
Foradetailedderivationofthetwoaboveequations,pleaseseeAppendixC.1.
pWositionsecanµalsobycomputemeansofthe(6.1)distancetobetweenthemultimodalpositionµMandtheunimodal
A/V|µM−µA/V|=σ2A/V|µA−µV|/σA2+σV2.(6.3)
78
dalMultimoIntegratedeachingT
Thatis,themultimodalandthereforetheteachingsignalislocatedmorecloselytothepositionof
themoreprecisemodality,e.g.,visionincomparisontoaudition,asshownhere.
Theaboveequations(6.1)to(6.3)illustratehowwecandescribeanintegratedmultimodal
mapataspecificpositioninspaceandtime.Nextwehavetotakeintoaccountthattheintegrated
multimodalmapwillprovidetheguidingsignalforunimodalmapformationandalignment.The
equationsofoptimalcombination,however,donotdescribehowunimodalmapschangeduringthe
learningprocess.Mapformationandalignmentareaneffectofsynapticplasticity(mathematically
describedbySTDP),aswewilldiscussinSec.6.2.2.Inthissectionwefocusonfundamental
characteristicsofthemultimodalmapthatfollowdirectlyfromoptimalmultimodalintegration.
Fornow,wepresumethattheunimodalmapsthatformthemultimodalmaparevariableintime.
Moreprecisely,ontheonehand,thepositionoftheunimodalmapsµA/V(t)canandwillvary
duringthecalibrationprocess.Ontheotherhand,themapscanimproveinprecisionsuchthat
σA/V(t)decreasesaswell.ThesetwoeffectsleadtoavariableiMTwithtime-dependantµM(t)
andσM(t)duringthecalibrationprocessoftheunimodalmaps.
Inshort,themultimodalmapispositionallyflexibleoverthecourseofthealignmentprocess
becauseitisthecombinationoftwoormoresensorymaps(6.1)to(6.3)thatthemselveschange
duringthelearningprocess.Wewillnowdiscussingreaterdetailwhatpositionalflexibilitymeans.
Wi.e.,ehathevecmeanµharacterizedofitsfiringeachprofile.mapinThthisus,wpapeerfirstbycitshangestandσard(Fig.6.4deviationA)andσandseethatestimaatorsensoryposition,map
Vthatbecomesmoreprecise(blueprofile)“attracts”thepositionofthemultimodalmap,i.e.,the
learningsignal.SensorymapsthatincreaseinaccuracygetmorestablepositionallyduringsSTDP
learningprocesses.Second,ashiftingsensorymap,forinstance,vision(blueprofile)inFig.6.4B,
inducesashiftofthemultisensorymaptowardsthestationarysensorymap(yellowprofile).The
upshotofiMTevenwithoutincludingeffectsofsSTDPinthecontextofmapalignmentisalready
thatamorestaticsensorymapattractstheteachingsignalandwillthereforehardlyadaptatall.
Insummary,wehaveanalyzedcharacteristicsoftheoptimallyintegratedmultimodalmapand
havetherebyshownthreeessentialaspects.First,themultimodalmapisalwaysmoreprecisethan
anyunimodalmap;cf.(6.2).Second,theoptimalmultisensoryinstructivesignalislocatedmore
closelytothepositionofthemoreprecisemodality;see(6.3).Third,onthebasisofanadaption
ofthemonosensorymapsthroughSTDPlearningrules,(6.1)and(6.2)describehowposition
andprecisionofthemultimodalteachervariesforgivenµA/V(t)andσA/V(t)oftheinputmaps.
Equippedwiththesecharacteristicsoftheteachersignal,wenowanalyzehowthecharacteristicsof
iMTinfluenceunimodalmapadaptationonthelevelofsynapticplasticity,thatis,inphysiological
andmathematicaltermsforsSTDPmapalignment.
6.2.2HowdoiMTcharacteristicsinfluencemapadaptation?
Intheprevioussectionwecouldseethatanintegratedmultimodalteacherhasaspecificvariance
andpositionthatbothdependonthecharacteristicsoftheinputmaps;cf.(6.1)and(6.2).In
addition,thesecharacteristicswillvaryduringmapcalibration.Inthissection,wethereforediscuss
howteacherswithdifferentaccuraciesandteachersthatareshiftingduringthelearningphase
t.alignmenmapaffectTheresultsaspresentedinthefollowingarebasedonnumericalsimulationsofaninhibition-
mediatedSTDPmapalignmentprocess.Thebasicstructureofourmodelconsistsofthethree
moteacherdalitiesmoadalituyditionTviaA,STDPsomatosensorylearningSrules;,andseevisionFig.V6.5.thatTheadaptfundamentothetalmprinciplesultimodalofinhibition-inhibitory
mediatedSTDPmapalignmenthavebeendescribedelaboratelybyFriedelandvanHemmen[78].
Allsensorymapsareone-dimensionalmapsofNPoissonneurons[145].Eachmodalityconsistsof
aninputmapwithaGaussianfiringprofile
fiA/S/V(y):=SA/S/Vexp−(y−xi)2/2σ2A/S/V(6.4)
79
IntegratedMultimodalTeaching
Figure6.4:Characteristicsofanintegratedmultimodalmap.DuringtheiMTlearningprocessthe
differentmodalities(µA/Vchanges)alignwiththeinstructivesignalprovidedbythemultimodal
map,andtheirreceptivefieldsbecomemoreprecise(σA/Vdecreases).Giventhatthemultimodal
mapisacombinationoftheunimodalsensorymaps,itsprecisionandpositionaredeterminedby
thesensorymapsandconsequentlychangeaswell.Therefore,wevaryonemodality(blueprofile)
inprecision(A)andposition(B)andleavetheothermodality(yellowprofile)unchanged.The
resultingdevelopmentofthemultimodalteacher(redprofile)isdepictedintheuppergraphs.(A)
Themodality(blueprofile)thatgetsmorepreciseattracts(6.1)andsharpens(6.2)themultimodal
teacher(redprofile).(B)Inaddition,themodality(blueprofile)thatalignswiththeteacher(red
profile),inducesthemultimodalmaptoshiftaswell.Thedistancebetweenthemultimodalteacher
andthestationarymap(yellowprofile)thereforechangesaswell(cf.(6.3)).
for0≤i≤N,withstimuluspositiony,thei-thmapneuron’spreferredpositionxi,standard
deviationσA/S/V,andSA/S/Vbeingthemaximalamplitudeoftheactivity.Fortwodifferentmaps,
e.g.audition(A)andvision(V),theaccuraciesdifferandσA=σV.Theinputlayerprojectsvia
all-to-allsynapsesJjA/S/V(t)ontoneuronjofthemaplayer.Inaddition,eachneuronofthemap
layerreceivesinhibitoryinputfromonecorrespondingneuronoftheteachermodalityT.The
synapsesconnectingteacherandmaparemodeledasstaticone-to-onesynapticconnections.
artificialWeinteacterruptherthatthemisnotultimodalgeneratedfeedbacfromklotheopmultimocompareddaltorepresenFig.6.1tationandbut6.2shobywssimtheessulatingentialan
characteristicsthatwederivedinSec.6.2.1.Aneuronpoftheteachermapthereforerespondstoa
stimulusatpositionyinformofan“inverted”Gaussianfunction
fpT(y):=ST1−exp−(y−xpT)2/2σT2(6.5)
withxpbeingthepreferredpositionofneuronp,STbeingthemaximalfiringactivityoftheteacher
80
eachingTdalMultimoIntegrated
Figure6.5:Modelarchitecture.Amultimodalobjectsuchasawaterdrophittingtheskinproduces
sensoryinputforaudition,somatosensory,andvision.WemodeltheinputbymeansofaGaussian
fiA/S/V(y)soastostimulatejneuronioftheinputlayer.Neuronsoftheinputlayerprojectthrough
all-to-allsynapticconnectionsJ(t)toneuronsjofthemaplayer.Thesesynapticconnections
willadapttoaligntheoutputofA/Sthe/VmaplayertotheinhibitoryteacherinputbymeansofsSTDP.
HereweusethismodelsetuptostudytheconsequencesofaniMT,whichisthemultimodal
representationofallsensorymodalities.
neurons,andσTbeingthewidthoftheteacherinfluence.Theprocesswehavejustdescribed
mathematically,whereateachersuppressestheactivityinthemaplayersatlocationsdifferent
fromtherealobject’spositiony,iscalledselectivedisinhibition[78,127].
Toanalyzethealignmentprocessweperformnumericalexperiments.Weintroduceanartificial
shiftΔinthesensorypositionofthemodality,e.g.,realizedthroughprismsintheKnudsen
[158,159]experimentonthebarnowl.Inmathematicalterms,wereplaceybyy+Δin(6.5).
Thesensorymodalities,say,auditionA,somatosensoryS,andvisionV,subsequentlychangetheir
synapticconnectionsbetweentheinputandthemaplayertocompensate(follow)theshift;see
6.5.Fig.Thesituationinournumericalexperimentscorrespondstothesituationofshiftingvision
bylenses[158,159].AsexplainedinSec.6.3.1,ashiftofvisioninducesadisplacementofthe
multimodalteacher.Consequently,ashiftofallsensorymapswithrespecttothepositionof
thecombinedmultimodalteacheroccurs,thatis,againstintuition,lensesshiftingvisionhardly
displacevisionwithrespecttotheteacherbutdisplaceallothersensorymapswithrespecttoour
multimodalteacher.Therefore,thebiologicalexperimentsandournumericalexperimentsagree.
Thechangeswithinthesynapticconnectivitypatternduringthealignmentprocesscanbe
describedandexplainedbySTDP[78,145].Toquantifychangeswithinthesynapticpatternwe
introducethemapestimationerrorEasaleast-squareerror.Itmeasureshowwellamapcan
reproducepositionalinformationofmsystematicallyvariedincomingsignals,
m
E=xi−xiT
=0i
2/m.(6.6)81
IntegratedMultimodalTeaching
Here,xidenotesthemeanpositionofthefiringprofileofthemap,andxiTrepresentsthetheoretical
position.ThesmallerthevaluefortheerrorE,thebetterthemapreproducesthereal(theoretical)
positionofanobject.TheparametersusedinournumericalexperimentsaregivenintheTableC.1.
Multimodalteacherwithvaryingvariance
Oneoftheessentialcharacteristicsofthemultimodalteacherisitsabilitytoimproveinprecision
duringthealignmentprocessasaconsequenceofdynamicunimodalmaps;see(6.2).Afterbirth,
mostanimalsfirsthavetolearnhowtointerpretthesensoryinformationtheyreceive.Allavailable
mapsasgivenbygeneticsareveryinaccurate[181,271].Furthermore,WallaceandStein[279]
observedthatmultisensoryneuronsincatsyoungerthan28daysreliablyrespondtoinputsfrom
morethanasinglemodalitybutdonotshowmultimodalenhancementordepression.These
observations,however,aremeasurementsonthelevelofsinglecells.Subsequentconclusionsinthe
contextofneuronalmapsandestimatorsareelusive.Therefore,itisfundamentallynecessaryto
showthatlearningstillfunctionsoratleaststartswithaninaccurateteacher.Beforewestudy
theeffectofadynamicteacheronsSTDPmapalignmentwewillclarifyhowteacheraccuracy
isdefined.Wenotethatthelessaccuratetheunimodalmaps,thelargerthevarianceofthe
multimodalteacher(6.2),thatis,theestimatedpositionandthustheteachingsignalisspatially
lessaccurate.Consequently,wemodeltheteacheraccuracybyaddingaGaussiandistributed
randomshiftΔTtothecorrectteacherposition.Asaresultofthenumericalexperimentswitha
varyingteachershiftΔT,weseethattheunimodalmapsadaptmoreslowlyandwithalargermap
estimationerrortoalessaccurateteachersignal;cf.Fig.6.6.Withinagiventimewindow,an
inaccurateteacherthereforeslowsdownbutdoesnotpreventthemapalignmentprocess.Wecan
thereforeconcludethattheiMTconceptdoesnotdependcruciallyonaconcreterealizationofthe
tegration.indalultimom
Figure6.6:InfluenceofteacherprecisionΔTonmapalignment.Wedepictthemapestimationerror
E(6.6)formapalignmentwithamultimodalteachervaryinginprecision.Themapestimation
errorbetweenteacherpositionandrepresentedpositiononthemapdropsfasterandtoalower
levelwithamoreprecise(lowerΔT)teacher.Theprecisionoftheteacherthereforeinfluencesthe
temporaldevelopmentofmapalignment.
Multimodalteacherwithdynamicallychangingposition
AsillustratedinFig.6.4,theintegratedmultimodalteacherpositionshiftsduringthealignment
process.Therefore,wemodelteacheractivitiesthatcontinuouslyshiftinspaceandanalyzehow
theirshiftingvelocitiesinfluencemapalignment.Forthetimestept=2000s,Fig.6.7showsthe
temporaldevelopmentofsensorymaps,i.e.theirsynapticpattern,modifiedbymeansofteaching
82
IntegratedeachingTdalMultimo
signalsshiftingwithdifferentvelocities.Wenotethatthedynamicsoftheteacheressentially
determinestowhichdegreeamapcanadapttotheteacher.Theslowertheteacherpositionshifts,
thebetteramapcanadapttotheteacher.Moreover,thecriticalvelocityuptowhichmapadaption
worksproperlydependsontheprecisionofthesensorymodalitythatisreflectedintheprecisionof
theinputmap.Alessprecisemodalitycanstilladapttoateacherwithagivenvelocity,whereasa
precisemapcannotfollowanymore;cf.Fig.6.7,changefromtoptobottom.
perfFigureorm6.7:edwithInfluenceinputofdynstandardamicallycdeviationhangingσ=teac0.03herp(topositionsline),σon=map0.05(balignmenottom)t.Simandteaculationsher
standarddeviationσT=0.02.Fordetailsregardingtheparameters,seeAppendixC.3.Weshift
theteachersignal,thatis,theinverseGaussianprofile,inspace.Theredlinesindicatethemean
positionoftheteacher.Inournumericalexperimentstheteacherpositionstartsshiftingfromthe
simdashedulatedredtimeline(ofΔ=20000.1s.).TheThedistancestraightbredetwlineeentheindicatestworedhowlinesfarthedepteacendsheronhastheshifteddifferentwithinshiftingthe
vtheelocitdiagoyofnal.theTheteacboherxatsignal.thetopTheleftshosynapticwsthepatternsynapticofthepatternsensorycausedmapsbyareasloinitiallywlyloshiftingcatedteacaroundher.
toWiththearighcertaint,thedelateacy,herthevelosynapticcitypatterndoubles.ofAstheamapconsequehasnce,adaptedthetomaptheadaptsteachermucmap.hlessByunsteptilpingthe
velocityreachesacriticallevel(toprightbox)wherethemapdoesnotadaptatall.Thevelocity
oftheteacherthereforeessentiallydeterminestowhichdegreeamapadaptstotheteacher.This
effectcanbecompensatedbydecreasingmapprecisionasrealizedfromthetoptothebottomline.
Wmap.ethenTheseecriticalthatvtheelocitysynapticofpatternnon-adaptationofalessisthereforeaccuratemaphigherstillforaloadaptsw-pvreerycisionwellmaptothethanteacforhera
map.high-precision
meansWeofconcludesSTDP.thatHowevdifferener,ttheteacteacherheraccuraciesaccuracywillnotinfluencespreventhettempalignmenoraltdevofdifferenelopmenttmapsand,btoy
somedegree,theaccuracyofthefinalstate.Wehaveseenthattheshiftingvelocityoftheteacher
pflexibleositionthandeterminesanaccuratetowhicmhapbdegreeecauseaitmapcancanadaptadapttoatothfasteretedyacher.namicAallesslycaccuratehangingteacmapher.ismoreThe
precisionofthedifferentcontributingunimodalmapsthereforeseemstodeterminehowtheyadapt
totheteachersignal.Inthenextsectionwewillstudytheseeffectsindetail.
83
IntegratedMultimodalTeaching
6.2.3HowdoesiMTcalibratedifferentunimodalmaps?
Havinganalyzedfirstthecombinationofanintegratedmultimodalmapasitarisesfromunimodal
andmapslastandstepthe(seeeffectFig.ofa6.2)howdynamicallythecessentialhangingcteacharacteristicsheronofmapthealignmendifferentt,wmoesimdalities,ulateinintheparticular,third
thedifferentaccuraciesofthesensorysystems,influencemapformationandalignment.
ResultsofoursimulationsofmapformationandalignmentareshowninFig.6.8.Basedon
thesenumericalexperimentsweobservethatmapformationandalignmentbymeansoftheiMTis,
inofmapgeneral,pformationossibleandforinputsalignmenoftdepdifferenendtontheaccuracies.accuracyHowevofer,thethesensoryqualityandsystem.temInporalpassing,progresswe
notethatwithinthealignmentprocess,ahigh-precisionmapreachesamodifiedsynapticpattern
thatisagainmoreprecisethanthatofalow-precisionmap.Moreimportantly,ahigh-precision
mapadaptsmoreslowlytotheteacherinputthanamapwithlow-precision.Inconclusion,the
highertheprecisionofasensorymap,themorestaticthemap.Wewillseeinthenextsectionhow
thesefindingscanexplainexperimentsofsensorymapformationandadaptation.
plotsFiguresho6.8:wtheInfluencemapofestimationvaryingerrorinputEasmapgivenprecisbyion(6.6)(σ)fromonmaunimopdalformationmapsinandcomparisonalignment.toThethe
teacher.Themapsvaryinprecision(inputstandarddeviationσ)asindicated.Thegraphsshow
isthemapenlargedtoestimationclearlyerrorshowEforwhichmapmapformationadapts(A)faster.andforBothmapplotsshoalignmenwtthat(B).mapsThebwithoxedahigherregion
precisionadaptmoreslowlytotheteacherinputthanlower-precisionmaps,butregainhigher
precisionafterthemapcalibration.
6.3Modelresultsandexperimentalfindings
AfterdetailedanalysisandbeingequippedwithadeeperunderstandingoftheiMTconcept,we
returntotheexperimentspresentedintheintroductionandreviewtheminthelightofouriMT
concept.Indoingso,wedistinguishbetweenthepreviouslymentionedexperimentsthatcanbe
explainedbyavision-guidedconceptofmapcalibrationandthosethatcannot.TheiMTconcept
wepresenthereoffersexplanationsforboth.
Inthefollowingwewillanalyzeagroupofexperimentsthatmotivatedtheideaofvisionas
thedominantguidingsensorymodality:theprismglassexperimentsofKnudsenandcoworkers
[127,158].Wereviewtheremainingexperimentswhiledenyingavision-guidedmapalignment.We
thenillustrateusingexperimentalevidenceasexamplesfromSteinetal.[277,279]howaniMT
conceptcouldwork.Finally,wemakesuggestionsandtestablepredictionsfornewexperiments
concept.iMTtheonbased
84
eachingTdalMultimoIntegrated
AnimalσAσVµµVA−−µµMAµµVV−−µµAM
Human1◦0.02◦99.96%0.04%
Barnowl2◦0.3◦97.8%2.2%
Cat8◦0.2◦99.93%0.06%
Table6.1:Visualandauditorylocalizationcapabilitiesofhuman[57,194],barnowl[106,157],and
cancat[b19e,42].distinguishedThevisualinoneresolutionsdegreeofaretheoriginallyvisualfield.givenWeinhavcyclesepassumederdegree,thei.e.,resolutionshowmantobyelinesthe
inverseofthesevalues.Onthebasisofthesevarianceswecombinethetwounimodalmapstoa
multimodalteacherandcalculatethedistanceofeachmaptothemultimodalmapdueto(6.3)in
comparisontothedistancebetweenthetwomodalities.
6.3.1Experimentsprovision-guidedmapformation
Themostprominentexperimentssupportingvision-guidedmapformationaretheshiftingexper-
imentsbyKnudsenetal.[127,158].Here,prismsshiftthevisualsystemofowlsbyacertain
angle.Afteralearningperiodtheneuronalprojectionsfromtheauditorymaparerearrangedto
compensateforthemisalignmentbetweenvisualandauditorymaps;seeSec.1.3intheintroduction.
Thevisualmap,incontrast,remainsconstantduringthewholealignmentprocess.
CanouriMTconceptreproducethedominanceofvisionwithinthelearningprocess?Toanswer
thisteacherquestionmapaswegivhaenvebysim(6.3)ulated.Wenotealignmenthattofthemapsvisualwithsystemdifferentusuallyvarianceshasaanmducdhistancehigherstospatialthe
resolution(aboutafewarcminutes)thanallothersensorysystems,e.g.theauditorysystem
(aboutafewdegrees);seeTable6.1fordetails.Wenowrevisitsynapticplasticityduringaprism
shiftexperimentfromthepointofviewofiMT.
•Initialstate(Fig.6.9,top)
Auditoryandvisualmapsareshiftedwithrespecttoeachotherbecauseofanartificialshift
ofthevisualsystemthatisintroducedbyprisms.Giventhatthemultimodalteachermap
isacombinationoftheunimodalmaps,itsactivitymeanhasshiftedaswell.Anadaptation
processofallunimodalmapsisinduced.Againstnormalintuition,shiftingvisiondoesnot
onlyresultinashiftbetweenvisionandthemultimodalteachingsignal;itinsteadresultsina
shiftofallsensorymapscomparedtothemultimodalteachermap.Asaconsequenceofthe
greaterprecisionofthevisualsystem,themultimodalteachermapismainlydeterminedby
thecharacteristicsofthevisualmap,thatisσML≈σVandµML≈µV;cf.(6.1)and(6.2)).
Equation(6.3)neatlyillustratesthattheshiftbetweenvisionasthemostprecisesensorysystem
andthusthemultimodalteachermapissmallerthanforanyothersystem.
•Intermediatestate(Fig.6.9,middle)
Synapticconnectionstotheauditorymapstartchangingtocompensatefortheshiftbetween
auditionandthemultimodalteacher.BasedonourpreviousstudiesaspresentedinSec.6.2.2
and6.2.3,weknowthatbecauseofitshigherprecision,thevisualmapadaptsmuchmoreslowly
thantheauditorymap.Theauditorymapthereforestartsadaptingtothemultimodalteacher.
Theteacherasacombinationofthetwomodalitiesthusshiftstowardthevisualmap.The
auditorymapadaptsagaintotheteacherinputandhencecontinuestoshifttowardthevisual
map,whichstaysalmoststatic;seeFig.6.4,B.
•Finalstate(Fig.6.9,bottom)
Thetwomapshavebeenrealigned.Thepositionofthevisualmaphashardlychanged,whereas
85
86
IntegratedMultimodalTeaching
Figure6.9:Numericalrealignmentexperiments.Weillustratetheshiftingprocessfortheauditory
map(A)withvarianceσA=0.07andthevisualmap(V)withσV=0.03.Thedistancesbetween
teacTheherarrayandplotsunimoshodalwthemappsynapticositionsareconnectivitchosenypatternaccordinglyfortovision(6.3)(leftforthecolumn)correspandondingauditionvariances.(right
mapcolumn).A,theThevisualsynaptmapicV,strengthsandthehereteacbyherranmgeapbetareweensho0wnandin1.theThemiddleactivitycolumn.profilesTheofthecolorcoauditorydeis
specifiedinthebottomrightcorner.Theplotillustrateshowsynapticplasticitydependsonthe
faraccuracylessthanofathemolessdality.Inaccurateconcreteauditoryterms,maptheA.moreprecisevisualmapVshiftsmoreslowlyand
eachingTdalMultimoIntegrated
theauditorymaphasshiftedalmostthewholedistanceinducedbytheprismsatthebeginning
t.erimenexptheofKnInudsensumandmary,othersour[iMT127,158concept].Hocanwever,reproducesimilarthenumericaldifferentstepsresultsofcanthebeshiftingobtainedexpforerimenanytstwofo
systemarbitraryonlysensoryreflectsmapsthewithmoregeneraldifferingvconceptariancesof.aTherefore,dominancetheofacobsecurrvacyedfollodominancewingofnaturallythefvisualrom
concept.iMTthe
6.3.2Experimentscontravision-guidedmapformation
Intheintroductionofthischapterwereviewedsomeexperimentsthatquestionavision-guided
ofitslearninghighprocess.precisionThebutanalysisthatofvisionourisnotconceptstatic.hasshoItwncanbthatemovisiondifiedplabysyaadyndominanamticallyrolecbhangingecause
multimodalteachermapandthereforebyanyunimodalmap[68,77,202,249,258].
Furthermore,theiMTconceptshouldbeabletoreproduceformationofunimodalmapsin
lovisuallycalizesoudeprivndasedpreciselyanimals.asStudiesindividuals[152,231with,234]normalshowvision.thatblindMeasuremenhumans,tsofcats,WallaceandandferretsSteincan
[277,279]supportthecriticalviewofvision-dominatedlearningbyshowingthatreceptivefields
butalreadyisshrinkdominatedatbytimesvisionwheninnocasesvisualinwhichneuronsitisaarevailabpresenle.t.Learningispossiblewithoutvision
butInitthehascaonmuctexthoflowoureriMTprecision.conceptFaormationmultimoofdaltheteacotherhermapsmapiscanthereforeexiststillwithoutpaossiblevisualwithinmap,
certainlimits;cf.Fig.6.6and[146,158,271].
animal,Inthetherebfolloywingstudyingtext,wewhetherwillsomreproduceatosensoryformationandofauditoryunimomdalapsmapscandevinealopvisuallywithoutdeprivvisioned,
i.e.,withaveryimpreciseteacher.
•InitialSomatosensorystate(Fig.and6.10,auditorytop)mapsareonlycoarselyprewired,andthevisualmapdoesnot
existimpreciseatall.aswTheell;mseeultimoSec.dal6.2.2mapforasadetails.combinationofsomatosensoryandauditorymapsisvery
•Intermediatestate(Fig.6.10,middle)
Evwhileentheirwithvaveryariancesdecimpreciserease.teacThehermapearlierthesimauditoryulationsandillustratethatsomatosensorythelearningmapsstartprotcessowithimprovane
impreciseteacherstartsveryslowly,correspondingtothetopmostcurveinFig.6.6.Nevertheless,
twtheounimosensorydalmapsmapsasaimproveconsequence.slightly,asWithdoesthethemimproultimovingdalteacmapherthat(seeisFig.acom6.6),thebinationoflearningthe
accelerates.cesspro•Finalstate(Fig.6.10,bottom)
Thetwomapsarealignedandhavereachedtheirbest-possibleresolutions.
Together,multimodalteachingillustratesthattheiMTconceptisasuccessfulconceptfor
incalibratingparallel,diasffreperentortedunimoindalexpmaps.erimentsInsucourhinastrinsic[277,279].learningprocessallmapsdevelopandimprove
6.4DiscussionandOutlook
IntheapplicationpresenoftciMThaptertowmapefhaveormationintroandducedthealignmencontceptleadsoftointhreetegratedfundamenMultimotallydalnewTeacideas.hing.First,The
87
88
IntegratedMultimodalTeaching
Figure6.10:Numericalmapformationexperimentswithanimprovingteacher.Activityprofilesof
theauditorymapA(σA=0.03),thesomatosensorymapS(σS=0.045),andtheteachermapare
showninthemiddlecolumn.Theblurredteacherprojectsontothemapswithaprecisionthatstarts
atΔT=0.1andthenreduceslinearlyintimetoΔT=0.02fort=7500s.Theblackbarsbelow
thefiring-activityplots(middlecolumn)denotetheregionswithintheteacherisprojectingforthe
correspondingtimestep.ThearrayplotsshowthesynapticconnectionpatternJAforaudition
(leftcolumn)andJSforthesomatosensorymodality(rightcolumn).Thesynapticstrengthsrange
between0and1.Theinitialsynapticpatternisonlycoarselyprewiredanditisadditionallyblurred
bymeansofrandomlydistributedsynapsestrengths.Thecolorcodeisspecifiedinthebottomright
corner.Despiteanimpreciseteacher,bothmapscandevelopproperly.
eachingTdalMultimoIntegrated
theintegratedteachercontainsallavailableinformationofthemapsinvolved.Second,theteacher
isintrinsictothesystembecauseitisgeneratedonlyfrominformationavailablewithinthesystem
ofmaps.Third,asaconsequence,theteacheritselfisdynamicwiththepotencytoshiftinposition
andtoimproveinprecisionduringthelearningprocess.
Moreover,thepresentedresultsprovidethebasicinsightthatthemorepreciseasensorysystem
is,themoreslowlyitadaptstoinstructivesignals.Thisnaturalconsequence,whichwedenoteas
dominanceofaccuracy,explainsobservationswhereadominanceofvisionwithinthealignment
processmaybewronglyinterpretedaspurelyvisuallydriven.
state-of-the-artthetoarisonCompHere,wehavesimulatedmultimodalteachercharacteristicsformapcalibrationonthelevelof
spikingneuronsandsynapticplasticity,namely,sSTDP.Theteachersignalinfluencestheunimodal
mapsbyselectivedisinhibitionasproposedbyanincreasingnumberofexperimentalfindings,
forinstance,intheICC-ICX-OTpathway[14];seealsoSec.1.3oftheintroduction.Taken
together,ouriMTconceptthereforeconnectstwofundamentalneurophysiologicalphenomena:
inhibition-mediatedmapcalibrationandmultimodalintegration.
Previoustheoreticalstudieshavefocusedononlypartsoftheseaspects.GelfandandMysore
[86,204]proposedmodelsforHebbianmapcalibrationbasedontheassumptionthatvisionserves
asteachersignaltomodify,forinstance,synapsesoftheauditoryprojectionpathway.Asalready
pointedoutintheintroductionofthischapter,visionasteachersignalneverthelesshastobe
questioned.SteinandWallace[277,279]thereforesuggestedaconnectionbetweenmultimodalmap
andmapformationincats.ThelatestexperimentalstudiesonbarnowlsfromBerganandKnudsen
[14]alsoshowthatcross-modaleffectsexistintheICXandmayinfluenceexperience-dependent
calibrationofconvergingrepresentations.
Includingtheseexperimentalfindings,Wittenetal.[287]developedatheoreticalmodelfor
bimodalmapalignment.Intheirmodel,thesynapsesofboththeauditoryandthevisualmap
areadaptedduetoHebbianlearning.Bythesemeans,themodelcanreproducethedominanceof
visionsuchthatthechannelwiththeweakerorbroaderreceptivefieldsalwaysexhibitsmostorall
oftheplasticity.However,thetheoreticalmodelofWittenetal.isexclusivelybasedonexcitatory
inputsandratecodes.Thelatterassumptionsarebothquiteimplausibleinthecontextofmap
alignmentinICXandOT.Concretely,experimentaldatafromBerganandKnudsen[14]suggest
thatinstructivesignalstotheICXaregatedbyinhibition.Inaddition,Kempteretal.[145]
demonstratedthattheassumptionsunderlyingrate-basedHebbianlearningarenotnecessarily
valid,especiallyintheauditorysystemofthebarnowl.
modelofIncludingmapthealignmenlastmentbytionedmeansexpoferimenselectivtaleresults,disinhibitionFriedelonettheal.lev[78el]ofdevspikingelopedaneurons.theoreticalThe
modelshowsthataninhibitoryteacherisessentialtorealignalreadycalibratedunimodalmaps.
However,themodelonlycontainsplasticitywithintheauditorysystem,thatis,thevisualsystem
isstaticandprovidestheguidingsignalforauditoryspacealignment.
ThepresentiMTmodelcombinesthefundamentalaspectsofthemodelsfromWittenetal.
[287]andFriedeletal.[78]:spike-timing-dependentlearningbasedonbothselectivedisinhibition
andfullplasticitywithinallcontributingmodalities,thatis,allavailableinformationisintegrated
intoamultimodalmapasafirststep.Inasecondstep,themultimodalmapprovidesaninstructive
signalformapcalibrationtoallmodalities.Therefore,theiMTmodelcanexplainthedominant
roleofvisioninmapalignment,plasticityinmodalitieswithoutvision,plasticitywithinthe
visualmodalityguidedbyothermodalities,plasticityofnon-visualmodalitiesguidedbyanother
non-visualmodality,andanycombinationofthesesettings.
TestablepredictionsoftheiMTconcept
GiventhatwehaveshownthatouriMTconceptcanreproducebothexperimentssupportingand
questioningvision-dominatedlearning,itisnowtimetooffersomesuggestionsregardinghowthe
89
IntegratedMultimodalTeaching
iMTconceptcanbeexperimentallyverified.Todistinguishvision-dominatedteachingfromiMT
wehavetobypassthedominanceofvision.Wecaneitherstudyartificialsituationsinwhichthe
precisionofsensorymapsisequal,investigateanimalswithapoorlydevelopedvisualsystem,or
observemapformationwhilemultimodalmapsareexperimentallydeactivated.
Tofirstequalizetheprecisionofsensorymaps,onecould,forinstance,performshiftingexperi-
mentswithprismsthatdisplaceandblurthevisualinputatthesametime.Second,studyingmap
formationandalignmentforanimalswithamoreprecisesensorysystemthanvisioncouldalso
identifyifthereisageneralprinciplesuchasdominanceofaccuracy.Athirdpossibilitywouldbe
todeactivatethemultimodalintegrationduringthelearningprocess.GiveniMT,deactivationof
theandmalignmenultimotdalshouldmap(s)failincorrespthisond(s)case.toInacondeactivtrast,ationforaoftheteacvision-dominatedhingmodalitteachy.ingMapscenario,formatione.g.
vision-guidedmapalignment,nonegativeeffectwouldbeobservedonthelearningprocess.
okOutlo
iMTFiguresetup6.11:shoOutlownoink:Fig.iMT6.5netbwyorkmeanssetupofwithincludingfeedbacktheconmultimonections.dalinOnecantegrationextendprocesstheoncurrenthet
levelpreviouslyofspikinginSec.6.2neurons.andThatdepictedis,weinFig.automatically6.2.First,incorptheporatetheost-synapticfeedbackcurrenmectshαanismgenerateddiscussedby
itheincomingspikesPfromtheneuronsiofthedifferentsensorymapsA/S/Varelogarithmically
summedup;Ij=ilogαi.Second,theprobabilitytoevokeaspikeinneuronjofthemultimodal
jmapofthecanmbeultimomodeleddaltomap.relateThird,expweonenusetiallyantointheterneuroncompletlayeerinputto“invcurrenert”tIthejinjectedexcitatoryintoactivitneurony
profiletoaninhibitorydisinhibitionsignalthat,fourth,guidesmapcalibrationofthesensorymaps;
sensorypathwayandinputisfadedascomparedtoFig.6.5.Inotherwords,withthenetwork
setupdescribedabove,wecanneuronallycalculateanoptimalmultimodalmapemployingallthe
characteristicswehaveusedfortheiMTinthischapterbymultiplyingtheactivityprofilesof
thesensorymapsonthelevelofspikingneurons;fordetailsseeAppendixC.Wehavealready
inimplemenprincipletedpandossibletestedbutthisthesetupfeedbacandkmecthehanresultsismloneedsokfurtherpromising.Mapclarification.formationandalignmentis
Insummary,wehaveintroducedthenewideaofanintrinsicmultimodallearningprocessand
insodoinghaverevealedinterestingnewaspectsofmultimodalmapformation.Thequestion
of(multimodal)mapformation,however,isnotyetfullyunderstood.Onthetheoreticalside,
futuremodelsshouldincludeatwo-wayfeedbackbetweenunimodalmapsandthemultimodal
teacher.Wehavealreadyimplementedandstartedtotestamoresophisticatedmodelonthelevel
90
eachingTdalMultimoIntegrated
ofspiking-neurons;seenetworksetupinFig.6.11.Inshort,spikesfromthesensorymapsA/Vare
summedlogarithmicallyprovidingtheinputcurrenttotheneuronsofthemultimodalmapMthat
aremodeledtorespondexponentiallytotheinputcurrent.Thiscorrespondstoamultiplication
ofspikesandmodelsoptimalmultimodalintegrationonalevelofspikingneurons;see(C.6)in
AppendixC.Theactivityofthemultimodalteacheristhen“inverted”viainter-neuronsand
projectedbacktothemapneuronsofthesensorysystemsA/Vtoprovideaninhibitoryteaching
signal.WearelookingforwardtoderiveanexplicitmodelofiMTincludingfullneuronalfeedback
dynamicswithoutartificialnormalizationterms.
Ontheexperimentalside,experimentsmaywelldeliverconvincingevidencefororagainst
aconnectionbetweenmultimodalprocessingandmapcalibration.Itwillbeinterestingtosee
whetherasystemofmapssuchaswhatisfoundintheSC/OT(seeFig.1.4)isabletocalibrate
intrinsically,andwhethermultimodalmapsplayakeyroleincalibration.
InpresentingtheconceptofiMT,wehavenotonlyaddressedthequestionofhowglobal
structurescanenablelocalmechanismssuchassynapticplasticitytoformcomplexbrainstructures,
e.g.,theSC/OT,wehavealsogivenanotherlineofevidencethatsensoryexperienceisthedriving
forceforbraindevelopment.Togiveafinalstatementbeyondthischapter,thisisreallyremarkable
sincethebrainexiststoprocessthequantitywhichdeterminesitsfunctioning,viz.,sensory
experience.Thatiswhywebelieveitissoimportanttounderstandsensoryprocessing.
91
App
endices
A.Appendix:OptimalMapformation
A.1Arecipeformakingmaps
TobringtolifethemathematicalframeworkofSec.2.2,wepresentaneasystep-by-step“recipe”
tofindtheoptimalconnectivityinarealisticbiologicalsetup:
•First,wederivethetransferfunctionhix(t)thatdeterminestheresponseofthedetectoritoa
stimuluspulsethatoccurredttimeunitsagoatpositionx.
•Next,wecalculatetheFouriertransformHixofthetransferfunctionhix(t).
•Wechoosesuitablevaluesofτandσ.Ingeneralthenoise-to-signalratioτcanbeassumed
tobemuchsmallerthan1foranymeasurablesignal.Incontrast,σneedstobeestimatedin
dependenceuponthesituationathand[76,75,252].
•WethencalculatethematrixentriesMijasgivenbyEq.(2.19)andinvertthemodelmatrix
.M•WemultiplytheinvertedmatrixM−1bythevectorHixsoastofindtheinputconnection
x.Lstrengthsi•Finally,wecalculatetheinverseFouriertransformofLixsoastofindtheconnectionstrengths
lix(t).
A.2Nonlinearitiesininformationprocessing
Thepresentedmodelassumesalinearrelationbetweenstimulusanddetectorresponse.Fora
numberofsensorysystems,however,wefindnon-linearitiesinthemappingprocess.First,the
transferfunctionhcanbeanon-linearfunctionh˜.Second,theneuronaldetectorresponsecanbe
non-linear,typicallylogarithmic[169,213,172,136,53].Incaseofalogarithmicresponseanda
non-lineartransferfunctionthebiologicaldetectorresponser˜hastoberewrittenfrom(2.1)as
tr˜i(t)=logdxdτsx(τ)h˜ix(t−τ).(A.1)
spaceall−∞Toapplyourmodelwefirsthavetoincorporateanadditionalcomputationalstepcancelingthe
logarithm.Inabiologicalsystemthiscanberealized,e.g.,byneuronswithexponentialfiring
behavior.Assumingsuchaneuronalstepr˜i(t)asin(A.1)reducesto(2.1)withanon-linearh˜.
Wecanlinearizeanon-lineartransferfunctionbyaredefinitionofthesignals→s˜.Thatis,we
identifyappropriatecharacteristicsofthestimulusthatarelinearlyrelatedtor.Forexample,
insteadoflookingattheheatdistributionT(x,t)wecanconsidertheintensitydistributionofthe
correspondingradiation∼T4(x,t)duetotheStefan-BoltzmannLaw.Inthisway,areasonable
redefinitionofdetectorresponseandsignalcanallowforanoptimallinearstimulusreconstruction.
95
A.Appendix:OptimalMapformation
A.3Remainingderivationstepsleadingto(2.16)
Inthefollowingweelaboratesomestepsskippedinthederivationof(2.16)inthemaintext.In
doingsowetakeadvantageofideasduetothecalclulusofvariations[138].Wethereforestartfrom
(2.6)asaconditiontominimizetheexpectationvalueofthequadraticerrorwithrespecttothe
optimalreversetransferfunctions,theconnectionstrengthsljx(t).Thisleadsto
∂[sx(t)−sˆx(t)]2
∂ljx(t)=0foreveryj.
⇔[sx(t)−sˆx(t)]∂sˆxx(t)=0.(A.2)
∂lj(t)
Inordertosolve(A.2),weexpandtheestimatesˆx(t)usingEqs.(2.3)and(2.4)fromthemaintext
givingsˆx=χilix+χiλix+dysyhiylix+syhiyλix+syηiylix
i+syηiyλix+ξyhiylix+ξyhiyλix+ξyηiylix+ξyηiyλix.(A.3)
Thevariationofsˆin(A.3)leadsto
∂∂lsˆx((tt))=χj+dysyhjy+syηjy+ξyhjy+ξyηjy(0).(A.4)
x
jAsbefore,weassumeontheonehandthatallnoisetermsaswellastheexpectationoftheinput
arestochasticallyindependentofeachother.Ontheotherhandweusethefactthatallnoise
termshavezeromeanandthesignalisself-averaging.Withthesetwoassumptions,theexpectation
valuesspartialˆs/∂landsˆ∂ˆs/∂lfrom(A.2)canbewritten
sx(t)∂sˆxx(t)=dysx(t)(syhjy)(0)(A.5)
∂lj(t)
andsˆx(t)∂∂lsˆx((tt))=(χilix)(t)χj(0)+dydy(syhiylix)(t)(syhjy)(0)
x
ji+(syηiylix)(t)(syηjy)(0)+(ξyhiylix)(t)(ξyhjy)(0)
+(ξyηiylix)(t)(ξyηjy)(0).(A.6)
Inthenextstepwesubstitutethecorrelationtermsasgivenin(2.10)ofthemaintext.Toillustrate
thecalculations,whichsimplify(A.5)and(A.6),weanalyzetwoisolatedtermsfrom(A.6)asan
example.Theothertermsaretreatedinasimilarway.Wefirstsimplify
dydy(syhiylix)(t)(syhjy)(0)
i=dydydτdτdτsy(t−τ−τ)hiy(τ)lix(τ)sy(−τ)hjy(τ).(A.7)
i
96
A.Appendix:OptimalMapformation
Exploitingthecorrelationassumptionsthisexpressionbecomes
µs2dydydτdτdτδ(y−y)δ(t−τ−τ+τ)hiy(τ)lix(τ)hjy(τ)
i=µs2dydτdτhiy(t−τ+τ)lix(τ)hjy(τ)
iyy=µs2dy((hilix)◦hj)(t)(A.8)
iwiththeopencircle◦denotingtheautocorrelationintegraldefinedin(2.13).Wefocusonthe
secondtermintheright-handsideof(A.6).Thatis,
dydy(syηiylix)(t)(syηjy)(0)
i=dydydτdτdτsy(t−τ−τ)ηiy(τ)lix(τ)sy(−τ)ηjy(τ),(A.9)
iwhichsimplifiesto
µs2ση2maxdydydτdτdτδ(y−y)δ(t−τ−τ+τ)δijδ(y−y)δ(τ−τ)lix(τ)
<yy||i0<τ<tmax
=µs2ση2maxdydτdτδ(−τ+t)ljx(τ)
|y|<ymax
<t<τ0=µs2ση2maxdydτljx(t).(A.10)
0|<τy|<y<tmax
Altogetherthefinalexpressionsfortheexpectationvaluesbecome
∂sˆx(t)
sx(t)x=µs2hjx(−t)(A.11)
∂lj(t)
andsˆx(t)x=σχ2ljx(t)+ση2(µs2+σξ2)maxdydτljx(t)+(µs2+σξ2)dy(hiylix)◦hjy(t).
∂sˆx(t)
∂lj(t)0|y<τ|<y<tmaxi
(A.12)Equation(A.2)thereforetransformsinto
maxljx(t)σχ2+(µs2+σξ2)|y|<ymaxdydτση2(A.13)
<t<τ0+(µs2+σξ2)dy(hilix)◦hj(−t)(A.14)
yy
i=µs2hjx(−t).(A.15)
InsertingtheparameterσandτandapplyingaFouriertransformationfinallyleadsto(2.16).
97
A.Appendix:OptimalMapformation
Inordertotestwhethertheextremumisindeedaminimum,wehavetocalculatethesecond
readshwhicariation,v∂2[sx(t)−sˆx(t)]2
∂ljx(t)2
∂x∂sˆx(t)x∂sˆx(t)
=2∂ljx(t)s(t)∂ljx(t)−sˆ(t)∂ljx(t)
=0+2σχ2+ση2(µs2+σξ2)|y|<ymaxdydτ+
0<τ<tmax
(µs2+σξ2)dydτhjy(τ)2.(A.16)
Sincethesquaresarepositive,soisthesecondderivative,andthustheextremumisaminimum.
signalblurredGaussianA.4Inthissubsectionwepresentanequationequivalentto(2.16)ofthemaintextsoastoderivean
expressionforaGaussianblurredsignal.Asin(A.17),arealisticsignalwouldfulfillsomekindof
Gaussianrelationfortheexpectationvalue
sx(t)sx(t)
22
=Aexp−|x−x|exp−|t−t|.(A.17)
2σx22σt2
Forthiscasewecan,analogouslytoappendixA.3,deriveanequationlike(2.16).Sinceforthe
signaltheGaussiancorrelations,however,replacethedeltafunctions,e.g.,in(A.10),integralsover
spaceandtimecannotbeevaluateddirectly.Insteadtheycanonlyberestrictedtotheregion
wheretheGaussianisnon-negligible.Denotingthesetemporalandspatiallimitsbytandwe
canderivetheanalogueto(2.16),viz.,
22
ddtAexp−||exp−|t|2hjx+(t+t)
2σx22σt
=σχ2ljx(t)+ση2σξ2maxdydτljx(t)+σξ2dy(hiylix)◦hjy(t)
0|y<τ|<y<tmaxi
t2
+ση2|y|<ymaxdyddτAexp−2σ2ljx(t+t)
tmax<t<τ0+dyddtAexp−||2exp−t2((hiylix)◦hjy+)(t+t).(A.18)
22
i2σx2σt
Theeffectoftheadditionalremainingspatio-temporalintegralsascomparedto(2.16)isasmoothen-
ingofthefinalreconstruction.Notonlyisthevalueataspecificpointinspaceandtime(y,t)
takenintoaccountbutneighboringpointsinanearlyareasurroundingitareincludedaswell.
98
A.Appendix:OptimalMapformation
PseudoinverseA.5ApseudoinversematrixBofanarbitrarym×nmatrixAwithitselements∈R,Cisageneralization
oftheinversematrix.FollowingMooreandPenrose[201,223]thepseudoinverseBisdefinedby
equations,four
ABA=A,(A.19a)
BAB=B,(A.19b)
(AB)T=AB,(A.19c)
(BA)T=BA,(A.19d)
wheretheoverbardenotesthecomplexconjugation.ThepseudoinverseisusedforamatrixA
thatisofincompleterangandthereforecannotbeinverteddirectlybutonlyduetoauxiliary
constructions(A.19a,b).Thelasttwoequations(A.19c,d)tellusthattheproductofmatrixA
withitspseudoinverseBisHermitian.
Oneofthemostfamousapplicationsofthepseudoinverseistocalculatetheleast-squaresolution
ofasystemoflinearequations.
Ax=b→xˆ=Bb(A.20)
GivenamatrixAandavectorb,theabovesolutionxˆ=BbminimizestheEuclideannorm
||Axˆ−b||.Iftheinversematrixexists,thepseudoinversereducestothenormalinversematrix.
Forageneralreviewandapplicationsthereaderisreferredto,e.g.,[3].
99
B.Appendix:Lateral-linePerception
Incalculatethefollothewing,twweo-dimensionalbrieflyenlighflowteninathe(pretendedx,y)-planeparadox:whereasWhywecanwcannoteusecalculatecomplexanalysisanalyticallyto
withonlyrotational-symmetrictwocoordinatessettings,involve.g.ed.flowaroundbodies,eventhoughthiscorrespondsalsotoasetup
B.1Planepotentialflow
Wedefinethepotentialflowasasettingwheretheflowfieldvcanbecalculatedbymeansofthe
velocitypotentialΦ:Rn→Rthroughv=Φ.Inthesesettingsrotv=0;theflowfieldisvortex
free.Aspecialcaseofpotentialflowistheplanepotentialflow.Here,oneoftheflowfieldcom-
ponentsv=(vx,vy,vz)iszero,e.g.vz=0.Inaddition,theremainingtwocomponentsare
independentofthecoordinatewhichcorrespondstotheflowfieldcomponentthatiszero,e.g.,
v=(vx(x,y,t),vy(x,y,t),0).Incaseofplanepotentialflow,thefollowingstatementsholdtrue
forthepotentialΦandthestreamfunctionΨ;thelatterisdefinedbymeansofvx=∂Ψ/∂yand
vy=−∂Ψ/∂x,
∂2Φ∂2Φ∂2Ψ∂2Ψ
∂x2+∂y2=0=∂x2+∂y2.(B.1)
Thatis,boththepotentialΦandthestreamfunctionΨsolvetheplaneLaplacianequationand
bothareconnectedthroughtheCauchy-Riemanndifferentialequations,
∂Φ∂Ψ∂Φ∂Ψ
∂x=∂yand∂y=−∂x.(B.2)
Forthatreason,wecandefineacomplexpotentialF(z):=Φ(x,y)+iΨ(x,y)withx+iy=z∈C
wherewecanderivethecomplexvelocityw(z),
dzdF=w(z)=vx(x,y)−ivy(x,y).(B.3)
Aswecanseeabove,itisalsocomposedofrealvaluedfunctions,namely,vxandvy.Indoingso,
wecananalyticallysolvemanyhydrodynamicproblemssuchastheflowaroundprofiles.
B.2Rotational-symmetricpotentialflow
toThevariouscomplexhydrodescriptiondynamicofhproblems.ydroItdynamicisobsettinviousgsisthatquitonlyeptowwerfulaso-dimensionalitprovidesissuesancanbanalyticaleaddressedaccess
withcompaosedoftwone-dimensionalorealcocomplexordinatesfux,ynctionorr,φ.sinceInthetuitivelycomplex,onecocanordinatethinkofz=xextending+iy=rtheexp(iφcomplex)is
formalismtothree-dimensionsincaseofsymmetriesthateffectivelyreducethethree-dimensional
problemdemonstrateatbhandelowtoaexplicitlytwforo-dimensionalrotational-symmetricsetup.Howevfloer,w.itisnotpossibletodosoaswewill
101
B.Appendix:Lateral-linePerception
Thevelocityfieldforrotational-symmetricflowisgivenbyv=(vr(r,z,t),vz(r,z,t),0).The
continuityequationdivv=0nowreads
1∂(rvr)+∂cz=0.(B.4)
z∂r∂rThecontinuityequationholdstrueforastreamfunctiondefinedbyrvr=−∂Ψ/∂zandrvz=
.r/∂Ψ∂Sincewearestudyingpotentialflowtheflowfieldhastobevortexfree,viz.,rotv=0.Thetwo
components(rotv)and(rotv)ofthecorrespondingvorticityrotvareobviouslyzero.According
tosymmetryreasons,zhowever,rthethirdcomponent(rotv)φisnotobviouslyzero.Thatis,wehave
toclaim2
(rotv)φ=∂∂vzr−∂∂vrz=−r1∂∂zΨ2+∂∂rr1∂∂rΨ=0.(B.5)
tIfwowecomequationsbineforthebconothtintheuitvyelocityequationpoten(B.4)tialΦandandrotvthe=str0eam(B.5)functionsimilarlyΨ,to(B.1)wecanderive
2222∂∂r2Φ+r1∂∂rΦ+∂∂z2Φ=0=∂∂r2Ψ−r1∂∂rΨ+∂∂z2Ψ.(B.6)
Aswecanseethelefthandsideandtherighthandsideof(B.6)arenotofthesametype.The
streamfunctiondoesnotfulfilltherotational-symmetricLaplacianequation,i.e,ΔΨ=0.
Inaddition,thestreamfunctionΨandthevelocitypotentialΦarenotconnectedthroughthe
Cauchy-Riemanndifferentialequations(B.2)but
∂Φ1∂Ψ∂Φ1∂Ψ
∂r=−r∂zand∂z=r∂r.(B.7)
ThusimaginarywepartcannotofinterpretanalyticalthefunctionstreamF(z)function=Φ(Ψx,y)and+itheΨ(x,vyelo),citinypconcreto,otentialofΦtheasthecomplexrealvandelocitthye
potentialF(z).
Insummary,thestreamfunctionΨandthevelocitypotentialΦdonotsolvethesamedifferential
equation(B.6)–theLaplacianequation–and,inaddition,theyarenotconnectedbytheCauchy-
Riemanndifferentialequations.Againstintuition,wethereforecannotusecomplexanalysisto
describerotational-symmetricflowandthusthree-dimensionalsetupsevenincaseofthisspecial
.symmetry
102
iMTendix:AppC.
C.1Optimalcombinationoftwomodalities
Thefiringprofileofaneuronalmapcanbeinterpretedasthelikelihoodoffindinganobjectata
specificposition;seeSec.1.2intheintroduction.Inotherwords,neuronalmapsencodeposition
estimatorsbymeansoftheirfiringprofiles.Amodality,forinstance,auditionAorvisionV,can
berepresentedthroughGaussianfiringprofiles
fA/V(x):=SA/Vexp−(x−µA/V)2/(2σ2A/V)(C.1)
withmappositionxandestimatedposition,i.e.,meanµA/V,standarddeviationσA/V,and
maximalamplitudeSA/Voftheactivityprofiles.Tocombinetwosuchestimatorstoamultimodal
estimateµMwehavetocalculateaweightedsumofthemeanvaluesµVandµAleadingto
µM=gVµV+gAµA(C.2)
withthenon-negativeweightsgVandgAfulfillinggV+gA=1.Thevarianceσ2Mofthecombined
readsestimatorσ2M=gV2σV2+(1−gV)2σA2.(C.3)
Next,wehavetochoosetheweightgVsoastominimizethevarianceσ2MwithrespecttogV;that
is,∂σM/∂gV=0.SolvingthelatterequationleadstogV=1−gA=σA2/(σA2+σV2)andfinallyto
σ2M=[σA2/(σA2+σV2)]2σV2+([σV2/(σA2+σV2)]2σA2=σV2σA2/(σA2+σV2).(C.4)
Wecaninserttheaboveresultinto(C.2)andcalculatetheoptimalcombinationscheme
µM=σA2/(σA2+σV2)µV+σV2/(σA2+σV2)µA.(C.5)
Formathematicaldetailsregardingthederivationseeelsewhere[276].
Theaboveformulaisabletodescriberesultsofneuronalprocessesbutwithoutexplaininghow
theyareachieved.Inalmostallcasesitispracticallyunknownhowtheneuronalsystemcomputes
anoptimalestimate.Inthecontextofneuronalmaps,however,asimplemultiplicationofthe
activityprofilesoftheinvolvedmapscanneuronallyrealizetheaboveoptimalcombinationscheme.
Themultimodalactivityprofileisthengivenby
fM(x)=SAexp−(x−µA)2/(2σA2)SVexp−(x−µV)2/(2σV2)
=SASVexp−1/2(µA−µV)2/σV2+σA2(C.6)
factorscaling2
exp−1/2σV2+σA2/σA2σV2x−σV2µA+σA2µV/σV2+σA2.(C.7)
1/σ2MµM
StandarddeviationσMandmeanµMofthemultimodalprofileareidenticaltotheoptimalvalues
wehavederivedabove;cf.(C.4)and(C.5).
103
iMTendix:AppC.
C.2Optimalcombinationofthreemodalities
WeextendtheoptimalcombinationschemefortwomodalitiesasderivedinSec.C.1tothree
modalities.ThemultimodalestimateµMofthreemodalities,say,auditionA,visionV,and
somatosensorySrepresentedbytheirmeanvaluesµA,µV,andµSwiththeirstandarddeviations
σA,σV,andσSisgivenby
σ2σ2σ2σ2σ2σ2
µM=VSµA+ASµV+AVµS,
σA2σV2+σA2σS2+σV2σS2σA2σV2+σA2σS2+σV2σS2σA2σV2+σA2σS2+σV2σS2
(C.8)
aconvexcombinationofµA,µV,andµS.ThecombinedmaximumlikelihoodestimateµMfeatures
thesmallestpossiblestandarddeviationσMgivenby
MσA2σV2+σA2σS2+σV2σS2
σ=σAσVσS.(C.9)
Thedistancebetweenthemultimodalestimateandaunimodalone,forinstanceaudition,isgiven
yb
22σA2σV2+σA2σS2+σV2σS2
t|µM−µA|=σA2σS|µA−µV|+σV|µA−µS|.(C.10)
FordistancesbetweenthemultimodalestimateandµVorµSonecaninterchangeAwithVorS.
104
AppC.iMTendix:
C.3ModelParameters
InTableC.1wesummarizethemodelparametersasusedinournumericalexperiments(seemain
text).
aluevparameternumberofmapneuronsN=100
synapseslearningminimalstrengthJmin=0.0
maximalstrengthJmax=0.25
maximalinitialstrength(J0)max=0.25
strengthofteachersynapsesJT=−1
timeonserespostsynapticptoinputspikeτTI=10ms
toteacherspikeτ=25ms
inputamplitudeS=50s−1
teacheramplitudeATA/V=/S100s−1
teacherwidthσT=0.025
shiftmaps–teacherΔ=0.2
learningtriallengthT=0.5s
simlearningulationtimparameterestepηΔt==3∙0.105−ms6
weightchangein
upupononinputoutputspikspikeewwout==1.−510.0
TableparametersC.1:ofMothedellearningparameterswindoaswusearedintakourennfromumerical[78]withexperimendifferingtsor(seemainadditionaltext).parametersShapeandas
e.voabindicated
105
Bibliography
Bibliography[1]Abbott,L.F.&Nelson,S.B.(2000)Synapticplasticity:tamingthebeast,NatNeurosci3
1178–1183.[2]Alais,D.&Burr,D.(2004)TheVentriloquisteffectresultsfromnear-optimalbimodal
integration,CurrBiol14257–262.
[3]Albert,A.(1972)RegressionandtheMoore-Penrosepseudoinverse(Academic,NewYork).
[4]Amemiya,F.,Goris,R.C.,Masuda,Y.,Kishida,R.,etal.(1995)Thesurfacearchitectureof
snakeinfraredreceptororgans,BiomedRes-Tokyo16411–421.
[5]Amemiya,F.,Nakano,M.,Goris,R.C.,Kadota,T.,etal.(1999)Ultrastructureofthe
crotalinesnakeinfraredpitreceptors:Semconfirmationoftemfindings,AnatRec254
107–115.[6]Amemiya,F.,Tatsuo,U.,Goris,R.C.,&Kusunoki,Y.T.(1996)Ultrastructureofthecrotaline
snakeinfraredpitreceptors:Semconfirmationoftemfindings,AnatRec246135–146.
[7]Anastasio,T.J.,Patton,P.E.,&Belkacem-Boussaid,K.(2000)Usingbayesruletomodel
multisensoryenhancementinthesuperiorcolliculus,NeuralComp121165–1187.
[8]Bakken,G.S.(2007)AcritiqueofthermalmodelingofsnakeinfraredreceptionbyJones,
LynnandStone(2001),JTheorBiol244167–168.
[9]Bakken,G.S.&Krochmal,A.R.(2007)Theimagingpropertiesandsensitivityofthefacialpits
ofpitvipersasdeterminedbyopticalandheat-transferanalysis,JExpBiol2102801–2810.
[10]Bartels,M.,M¨unz,H.,&Claas,B.(1990)Representationofthelaterallineandelectrosensory
systemsinthemidbrainoftheaxolotl,Ambystomamexicanum,JCompPhysiolA167
347–356.[11]Beauchamp,M.S.,Argall,B.D.,Bodurka,J.,Duyn,J.H.,etal.(2004)Unravelingmultisensory
integration:patchyorganizationwithinhumanSTSmultisensorycortex,NatNeurosci7
1190–1192.[12]Ben-Israel,A.&Greville,T.N.E.(2003)GeneralizedInverses:TheoryandApplications
(Springer,NewYork,NY),2ndedition.
[13]Benson,D.L.,Colman,D.R.,&Huntley,G.W.(2001)Molecules,mapsandsynapsespecificity,
899–909.2NeurosciRevNat[14]Bergan,J.F.&Knudsen,E.I.(2009)Visualmodulationofauditoryresponsesintheowl
inferiorcolliculus,JNeurophysiol1012924–2933.
[15]Bi,G.Q.(2002)Spatiotemporalspecicityofsynapticplasticity:cellularrulesandmechanisms,
319–332.87ernCybBiol[16]Bi,DepG.Q.endence&Ponoo,spikeM.M.timing,(1998)synapticSynapticstrength,modificationsandpinostsynapticculturedcellhipptypoe,JcampalNeuroscineurons:18
472.464–1010[17]Bi,G.Q.&Poo,M.M.(2001)Synapticmodificationbycorrelatedactivity:Hebb’spostulate
revisited,AnnuRevNeurosci24139–166.
[18]Bialek,W.,Rieke,F.,deRuytervanSteveninck,R.R.,&Warland,D.(1991)Readinga
neuralcode,Science2521854–1857.
[19]Bisti,S.&Maffei,L.(1974)Behaviouralcontrastsensitivityofthecatinvariousvisual
meridians,JPhysiol241201–210.
[20]Blauert,J.(1999)SpatialHearing(MITPress,Cambridge,MA).
[21]Bleckmann,H.(1994)Receptionofhydrodynamicstimuliinaquaticandsemiaquaticanimals
her).(Fisc[22]Boll,S.F.(1979)Suppressionofacousticnoiseinspeechusingspectralsubtraction,IEEET
AcoustSpeech27113–20.
[23]Braucdynamicshi,S.,andOrio,Pelectroph.,&Latorre,ysiologicalR.analysis(2004)ofCluescoldtoreceptorunderstandingtrpm8,ProcoldcNatlsensation:AcadSciThermo-USA
499.494–1515101
107
Bibliography
[24]Braun,H.A.&Hensel,H.(1980)Staticanddynamicdischargepatternsofburstingcold
[25]fibBraun,ersrelatedH.A.,Sctoh¨hafer,ypK.,othetical&Wissing,receptorH.mec(1984)hanisms,TheorienPflugundArchMoEurdelleJPhzumy¨ub3861–9.ertragungsver-
haltenthermosensitiverRezeptoren,FunktBiolMed326–36.
[26]Braun,H.B.,Dewald,M.,Sch¨afer,K.,Voigt,K.,etal.(1999)Low-dimensionaldynamicsin
sensorybiology2:Facialcoldreceptorsoftherat,JComputNeurosci717–32.
[27]Braun,H.B.,Huber,M.T.,Dewald,M.,Sch¨afer,K.,etal.(1998)Computersimulationsof
neuronalsignaltransduction:Theroleofnonlineardynamicsandnoise,IntJBifurcatChaos
881–889.8[28]Braun,W.,Eckhardt,B.,Braun,H.A.,&Huber,M.(2000)Phase-spacestructureofa
thermoreceptor,PhysRevE626352–6361.
[29]Bregman,A.S.(1990)AuditorySceneAnalysis(MITPress,Cambridge,MA).
[30]Bresciani,J.-P.,Dammeier,F.,&Ernst,M.O.(2008)Tri-modalintegrationofvisual,tactile
andauditorysignalsfortheperceptionofsequencesofevents,BrainResBul75(6)753–
760.[31]Bresciani,J.-P.,Ernst,M.O.,Drewing,K.,Bouyer,G.,etal.(2005)Feelingwhatyouhear:
auditorysignalscanmodulatetactiletapsperception,ExpBrainRes162172–180.
[32]vanBrunt,B.(2000)Thecalculusofvariations(Springer,Heidelberg).
[33]Bullock,T.H.&Diecke,F.P.J.(1956)Propertiesofaninfra-redreceptor,JPhysiol134
47–87.[34]Buonomano,D.V.&Merzenich,M.M.(2004)Corticalplasticity:Fromsynapsestomaps,
AnnuRevNeurosci21149–186.
[35]B¨urck,M.,Friedel,P.,Sichert,A.B.,Vossen,C.,etal.(2010)Optimalityinmono-and
multisensorymapformation,BiolCybern1031–20.
[36]B¨urck,M.&vanHemmen,J.L.(2007)Modelingthecochlearnucleus:Asiteformonaural
echosuppression?,JAcoustSocAm122(4)2226–2235.
[37]B¨urck,M.&vanHemmen,J.L.(2009)Neuronalidentificationofsignalperiodicitybybalanced
inhibition,BiolCybern100261–270.
[38]Campbell,A.L.,deCockBunning,T.,Stone,M.O.,Church,D.,etal.(1999)Surface
ultrastructureofpitorgan,spectacle,andnonpitorganepidermisofinfraredimagingboid
snakes:Ascanningprobeandscanningelectronmicroscopystudy,JStructBiol126105–120.
[39]vonCampenhausen,C.,Riess,I.,&Weissert,R.(1981)Detectionofstationaryobjectsby
theblindcavefishAnoptichtysjordani,JCompPhysiolA143369–374.
[40]Carr,C.E.,Maler,L.,&Sas,E.(1982)Peripheralorganizationandcetralprojectionsofthe
electrosensorynervesingymnotiformfish,JCompNeurol211139–153.
[41]Carr,R.W.,Pianova,S.,McKemy,D.D.,&Brock,J.A.(2009)Actionpotentialinitiation
intheperipheralterminalsofcold-sensitiveneuronsinnervatingtheguinea-pigcornea,J
1249–1264.587ysiolPh[42]Casseday,J.H.&Neff,W.D.(1973)Psychophysicalstudiesofmonkeyvision:III.Spatial
luminancecontrastsensitivitytestsofmacaqueandhumanobservers,JAcoustSocAm
365–372.54(2)[43]Catania,K.C.,Leitch,D.B.,&Gauthier,D.(2009)Functionoftheappendagesintentacled
snakes(Erpetontentaculatus),JExpBiol213359–367.
[44]Claas,B.(1994)Removalofeyesinearlylarvalstagesalterstheresponseoftheclawedtoad,
[45]Clegg,XenopusJ.C.laevis,(1968)tosurfaceCalculuswaofves,PhvariationsysiolBeh(Oliv56erand423–428.Boyd,Edinburgh).
[46]Clifton,R.K.(1987)Breakdownofechosuppressionintheprecedenceeffect,JAcoustSoc
1834–1835.82(5)Am[47]deCockBuning,T.(1984)Atheoreticalapproachtotheheatsensitivepitorgansofsnakes,
509–529.111BiolTheorJ[48]deCockBuning,T.,Terashima,S.I.,&Goris,R.C.(1981)Crotalinepitorgansanalyzedas
108
Bibliography
warmreceptors,CellMolNeurobiol169–85.
[49]deCockBuning,T.,Terashima,S.I.,&Goris,R.C.(1981)Pythonpitorgansanalyzedas
warmreceptors,CellMolNeurobiol1271–278.
[50]Cohen,M.L.(1977)Measurementofthethermalpropertiesofhumanskin.Areview,JInvest
333–338.69Dermatol[51]Cohen,Y.E.&Knudsen,E.I.(1999)Mapsversusclusters:differentrepresentationsofauditory
spaceinthemidbrainandforebrain,TrendsNeurosci22128–135.
[52]Coombs,S.,Anderson,R.,Braun,C.B.,&Grosenbaugh,M.(2007)Thehydrodynamic
footprintofbenthic,sedentaryfishinundirectionalflow,JAcoustSocAm1221227–1237.
[53]StevCopenselli,Mand.,RoWebque,er-FecA.C.,hnerOlivlawseira,areR.F.,transfer&Kinoucfunctionshi,O.of(2002)excitablePhysicsmedia,ofPhpsycysRhophevEysics:65
901.60[54]´Cur˘ci´c-Blake,B.&vanNetten,S.M.(2006)Sourcelocationencodinginthefishlateralline
canal,JExpBiol2091548–1559.
[55]Dan,Y.&Poo,M.M.(2004)Spiketiming-dependentplasticityofneuralcircuits,Neuron44
23–30.[56]Davison,A.P.&Fr´egnac,Y.(2006)Learningcross-modalspatialtransformationsthrough
spiketiming-dependentplasticity,JNeurosci265604–5615.
[57]DeValois,R.,Morgan,H.,&Snodderly,D.(1974)Psychophysicalstudiesofmonkeyvision:
Iii.spatialluminancecontrastsensitivitytestsofmacaqueandhumanobservers,VisionRes
75–81.14[58]Dehaene,S.&Cohen,L.(2007)Culturalrecyclingofcorticalmaps,Neuron56384–398.
[59]Deneve,S.,Latham,P.E.,&Pouget,A.(1999)Readingpopulationcodes:aneuralimple-
mentationofidealobservers,NatNeurosci2(8)740–745.
[60]Deneve,S.,Latham,P.E.,&Pouget,A.(2001)Efficientcomputationandcueintegration
withnoisypopulationcodes,NatNeurosci4(8)826–831.
[61]Deneve,S.&Pouget,A.(2004)Bayesianmultisensoryintegrationandcross-modalspatial
links,JPhysiology-Paris98249–258.
[62]Dr¨ager,U.C.&Hubel,D.H.(1975)Responsestovisualstimulationandrelationshipbetween
visual,auditory,andsomatosensoryinputsinmousesuperiorcolliculus,JNeurphysiol38
690–713.[63]Ebert,J.&Westhoff,G.(2006)Behaviouralexaminationoftheinfraredsensitivityof
rattlesnakes(Crotalusatrox),JCompPhysiolA192941–947.
[64]Eckertf,E.R.G.&Carlson,WalterO.(1961)Naturalconvectioninanairlayerenclosed
betweentwoverticalplateswithdifferenttemperatures,IntJHeatMassTransf2106–120.
[65]Eckhorn,R.,Bauer,R.,Jordan,W.,Brosch,M.,etal.(1988)Coherentoscillations:A
mechanismoffeaturelinkinginthevisualcortex?,BiolCybern60121–130.
[66]Ephraim,Y.&Malah,D.(2004)SpeechEnhancementUsingaMinimumMean-SquareError
Short-TimeSpectralAmplitudeEstimator,IEEETAcoustSpeech321109–21.
[67]Ernst,M.O.&Banks,M.S.(2002)Humansintegratevisualandhapticinformationina
statisticallyoptimalfashion,Nature415429–433.
[68]Ernst,M.O.,Banks,M.S.,&B¨ulthoff,H.(2000)Touchcanchangevisualslantperception,
69–73.3NeurosciNat[69]Ernst,M.O.&B¨ulthoff,H.(2004)Mergingthesensesintoarobustpercept,TrendsCognSci
162–169.8[70]Evans,L.C.(1998)PartialDifferentialEquations,volume19ofGraduateSeriesinMathe-
matics(AmericanMathematicalSociety,Providence,RI).
[71]Faisal,A.A.,Selen,L.P.J.,&Wolpert,D.M.(2008)Noiseinthenervoussystem,NatRev
292–303.9Neurosci[72]Fiala,D.,Lomas,K.J.,&Stohrer,M.(1999)Acomputermodelofhumanthermoregulationfor
awiderangeofenvironmentalconditions:thepassivesystem,JApplPhysiol871957–1972.
109
Bibliography
[73]Flanagan,J.L.&Lummis,R.C.(1970)Signalprocessingtoreducemultipathdistortionin
smallrooms,JAcoustSocAm47(6)1475–81.
[74]Franosch,J.-M.P.,Lingenheil,M.,&vanHemmen,J.L.(2005)Howafrogcanlearnwhatis
whereinthedark,PhysRevLett95078106.
[75]Franosch,J.-M.P.,Sichert,A.B.,Suttner,M.D.,&vanHemmen,J.L.(2005)Estimating
positionandvelocityofasubmergedmovingobjectbytheclawedfrogXenopusandby
fish—Acyberneticapproach,BiolCybern93231–238.
[76]Franosch,J.-M.P.,Sobotka,M.C.,Elepfandt,A.,&vanHemmen,J.L.(2003)Minimalmodel
ofpreylocalizationthroughthelateral-linesystem,PhysRevLett91158101.
[77]Frens,M.A.&vanOpstal,A.J.(1998)Visual-auditoryinteractionsmodulatesaccade-related
activityinmonkeysuperiorcolliculus,BrainRes46211–224.
[78]Friedel,P.&vanHemmen,J.L.(2008)Inhibition,notexcitation,isthekeytomultimodal
sensoryintegration,BiolCybern98597–618.
[79]Furuya,K.&Kataoka,A.(2007)Robustspeechdereverberationusingmultichannelblind
deconvolutionwithspectralsubtraction,IEEETAudioSpeech151579–91.
[80]Galan,R.F.,Ermentrout,G.B.,&Urban,N.N.(2007)Reliabilityandstochasticsynchroniza-
tionintypeivs.typeiineuraloscillators,Neurocomputing702102–2106.
[81]Galan,R.F.,Fourcaud-Trocme,N.,Ermentrout,G.B.,&Urban,N.N.(2006)Correlation-
inducedsynchronizationofoscillationsinolfactorybulbneurons,JNeurosci263646–3655.
[82]Gancarz,G.&Grossberg,S.(1999)Aneuralmodelofsaccadiceyemovementcontrolexplains
task-specificadaptation,VisionRes393123–3143.
[83]Gardner,J.L.,Merriam,E.P.,Movshon,J.A.,&Heeger,D.J.(2008)Mapsofvisualspacein
humanoccipitalcortexareretinotopic,notspatiotopic,JNeurosci283988–3999.
[84]Geisler,C.D.(1990)FromSoundtoSynapse:PhysiologyoftheMammalianEar(Oxford
Oxford).Press,yersitUniv[85]Gelfand,I.M.&Fomin,S.V.(1963)Calculusofvariations(Prentice-Hall,EnglewoodCliffs,
NY).[86]Gelfand,J.J.,Pearson,J.C.,Spence,C.D.,&Sullivan,W.E.(1988)Multisensorintegration
inbiologicalsystems(IEEEPress,NewYork).
[87]Gerstner,W.,Kempter,R.,vanHemmen,J.L.,&Wagner,H.(1996)Aneuronallearning
ruleforsub-millisecondtemporalcoding,Nature38376–78.
[88]Gerstner,W.&Kistler,W.(2002)SpikingNeuronModels(CambridgeUniversityPress,
bridge).Cam[89]Gerstner,W.,Kreiter,A.K.,Markram,H.,&Herz,A.V.M.(1997)Neuralcodes:Firingrates
[90]andGillespie,beyond,B.W.,ProcMalvNatlar,AcadH.S.,Sci&USA94Florencio,12740–12D.A.F.741.(2001)Speechdereverberationvia
maximum-kurtosissubbandadaptivefilltering,ProcIEEEIntConfAcoustSpeechSig-
3701–4.6cesspronal[91]Gonzalez,R.C.&Woods,R.E.(2002)DigitalImageProcessing(PrenticeHall,UpperSaddle
edition.2ndNJ),er,Riv[92]Goris,R.C.,Nakano,M.,Atobe,Y.,Kadota,T.,etal.(2000)Nervouscontrolofbloodflow
microkineticsintheinfraredorgansofpitvipers,AutonNeurosci-Basic8498–106.
[93]Goris,R.C.&Nomoto,M.(1967)InfraredReceptioninOrientalCrotalineSnake,Comp
BiochemPhysiol23879–892.
[94]Goris,ReceptorsR.C.in&aTCrotalineerashima,SnakS.I.e,T(1973)rimeresurusCentralFlaRespvovionseridis,toJExpInfra-redBiolStim58ulation59–76.ofthePit
[95]Goulet,J.,Engelmann,J.,Chagnaud,B.P.,Franosch,J.-M.P.,etal.(2008)Objectlocalization
throughthelaterallinesystemoffish:Theoryandexperiment,JCompPhysiolA1941–17.
[96]Grace,M.S.,Woodward,O.M.,Church,D.R.,&Calisch,G.(2001)Preytargetingbythe
infrare-imagingsnakePython:effectsofexperimentalandcongenitalvisualdeprivation,
BehavBrainRes11923–31.
110
Bibliography
[97]Gracheva,E.O.,Ingolia,N.T.,Kelly,Y.M.,Cordero-Morales,J.F.,etal.(2010)Molecular
basisofinfrareddetectionbysnakes,Nature4641006–1012.
[98]Gray,C.M.,K¨onig,P.,Engel,A.K.,&Singer,W.(1989)Oscillatoryresponsesincatvisual
cortexexhibitinter-columnarsynchronizationwhichreflectsglobalstimulusproperties,Nature
334–337.338[99]Grossbadaptiveerg,sS.,accadicRobeyerts,emoK.,vemenAguilar,tconM.,trol&byBullosupceriork,D.colliculus,(1997)AJneuralNeuroscimodel24ofm9706–9725.ultimodal
[100]Grothe,B.,Covey,E.,&Casseday,J.H.(2001)Medialsuperioroliveofthebigbrownbat:
Neuronalresponsestopuretones,amplitudemodulations,andpulsetrains,JNeurohysiol86
2219–2230.[101]Grothe,B.&Neuweiler,G.(2000)Thefunctionofthemedialsuperioroliveinsmallmammals:
[102]Gu,tempY.,oralAngelaki,receptiveD.E.,fieldsin&auditoryDeAngelis,analysis,G.C.J(2008)CompNeuralPhysiolAcorrelates186of413–423.multisensorycue
integrationinmacaqueMSTd,NatNeurosci111201–1210.
[103]Gutfreund,Y.,Zheng,W.,&Knudsen,E.I.(2002)Gatedvisualinputtothecentralauditory
1556–1559.297Sciencesystem,[104]Hafed,Z.M.,Goffart,L.,&Krauzlis,R.J.(2008)Superiorcolliculusinactivationcausesstable
offsetsineyepositionduringtracking,JNeurosci288124–8137.
[105]H¨ansel,E.&Schmidt,G.(editors)(2004)AcousticEchoandNoiseControl–APractical
Approach(Wiley,Hoboken).
[106]Harmening,W.M.,Nikolay,P.,Orlowski,J.,&Wagner,H.(2009)Spatialcontrastsensitivity
andgratingacuityofbarnowls,JVision9(7)1–12.
[107]Harris,G.G.(1964)Considerationsonthephysicsofsoundproductionbyfishes,inW.N.
Tavolga(editor),MarineBio-Acoustics,pp.233–247(PergamonPress,Oxford).
[108]Harris,G.G.,Flanagan,J.L.,&Watson,B.J.(1963)Binauralinteractionofaclickwitha
clickpair,JAcoustSocAm35672–678.
[109]Hartline,P.H.,Kass,L.,&Loop,M.S.(1978)MergingofmodalitiesintheopticTectum:
Infraredandvisualintegrationinrattlesnakes,Science1991225–1229.
[110]Hasan,K.M.,Salahuddin,S.,&Khan,R.M.(2004)Reducingsignal-biasfromMADestimated
noiselevelforDCTspeechenhancement,SignalProcess84151–62.
[111]Hassan,E.l.-S.(1985)Mathematicalanalysisofthestimulusforthelaterallineorgan,Biol
23–36.52ernCyb[112]Hebb,D.O.(1949)Theorganizationofbehavior:aneuropsychologicaltheory(Wiley,New
ork).Y[113]Helbig,H.B.&Ernst,M.O.(2007)Optimalintegrationofshapeinformationfromvisionand
touch,ExpBrainRes179595–606.
[114]Helmholtz,H.(1879)DieThatsacheninderWahrnehmung(AugustHirschwald,Berlin).
[115]vanHemmen,J.L.(2001)Theoryofsynapticplasticity,inFMoss&SGielen(editors),
HandbookofBiophysics,pp.771–823(Elsevier,Amsterdam),4thedition.
[116]vanHemmen,J.L.(2002)Themapinyourhead:Howdoesthebrainrepresenttheoutside
world?,ChemPhysChem3291–298.
[117]Hensel,H.(1975)StaticanddynamicactivityofwarmreceptorsinBoaconstrictor,P߬ugers
191–199.353hivArc[118]Hensel,H.&Wurster,R.D.(1970)Staticpropertiesofcoldreceptorsinnasalareaofcats,J
271–275.33ysiolNeuroph[119]Herz,A.V.M.,Gollisch,T.,Machens,C.K.,&Jaeger,D.(2006)ModelingSingle-Neuron
dynamicsandComputations:ABalanceofDetailandAbstraction,Science31480–85.
[120]Hilis,J.M.,Ernst,M.O.,Banks,M.S.,&Landy,M.S.(2002)Combiningsensoryinformation:
Mandatoryfusionwithin,butnotbetween,senses,Science2981627–1630.
[121]Hisajima,T.,Kishida,R.,Atobe,Y.,Nakano,M.,etal.(2002)Distributionofmyelinated
andunmyelinatednervefibersandtheirpossibleroleinbloodflowcontrolincrotalinesnake
111
Bibliography
infraredreceptororgans,JCompNeurol449319–329.
[122]H¨blindotting,hK.,umans:R¨Anosler,evF.,en&R¨t-relatedoder,pB.oten(2004)tialstudyAltered,ExpBrainauditory-tactileRes159interactions370–381.incongenitally
[123]Hubel,D.H.&Wiesel,T.N.(1977)Functionalarchitectureofmacaquemonkeyvisualcortex,
ProcRoySocLondB1981–59.
[124]Huber,M.T.&Braun,H.A.(2006)Stimulus-responsecurvesofaneuronalmodelfornoisy
subthresholdoscillationsandrelatedspikegeneration,PhysRevE730419291–04192910.
[125]opticsHughes,andA.retinal(1977)Thetoporganization.,ographyinofF.visionCrescitelliinmammals(editor),ofconHandbtrastingookoflifestsensoryyle:physiComparativologye,
volumeVII/5,chapter8,pp.613–756(Springer,BerlinHeidelbergNewYork).
[126]H¨urlimann,F.,Kiper,D.C.,&Carandini,M.(2002)Testingthebayesianmodelofperceived
speed,VisionRes422253–2257.
[127]Hyde,P.S.&Knudsen,E.I.(2001)Atopographicinstructivesignalguidestheadjustmentof
theauditoryspacemapintheoptictectum,JNeurosci218586–8593.
[128]Ito,M.(1989)Long-termdepression,AnnuRevNeurosci1285–102.
[129]Itti,L.&Koch,C.(2001)Computationalmodelingofvisualattention,NatRevNeurosci
194–203.2(3)[130]J¨arvilehto,M.(1985)Theeye:Visionandperception,inG.A.Kerkut&L.I.Gilbert(editors),
NervousSystem:Sensory,volume6ofComprehensiveInsectPhysiology,Biochemistry,and
Pharmacology,chapterFINDCHAPTER,pp.355–429(PergamonPress,Oxford).
[131]Jazayeri,M.&Movshon,J.A.(2006)Optimalrepresentationofsensoryinformationbyneural
populations,NatNeurosci9690–696.
[132]Jeffress,L.A.(1948)Aplacetheoryofsoundlocalization,JCompPhysiolPsychol4135–39.
[133]Jiang,B.,Trevi˜no,M.,&Kirkwood,A.(2007)Sequentialdevelopmentoflong-termpotentia-
tionanddepressionindifferentlayersofthemousevisualcortex,JNeurosci279648–9652.
[134]Jiang,W.,Wallace,M.T.,Jiang,H.,Vaughan,J.W.,etal.(2001)TwoCorticalAreasMediate
MultisensoryIntegrationinSuperiorColliculusNeurons,JNeurophysiol85506–522.
[135]Johnson,D.H.&Dudgeon,D.E.(1993)ArraySignalProcessing:ConceptsandTechniques
(Prentice-Hall,UpperSaddleRiver,NJ).
[136]Johnson,K.O.,Hsiao,S.S.,&Yoshioka,T.(2002)Neuralcodingandthebasiclawof
psychophysics,Neuroscientist8111–121.
[137]Jones,B.S.,Lynn,W.F.,&Stone,M.O.(2001)Thermalmodelingofsnakeinfraredreception:
Evidenceforlimiteddetectionrange,JTheorBiol209201–211.
[138]Jost,J.&Li-Jost,X.(1998)Calculusofvariations(CambridgeUniversity,Cambridge).
[139]Kaas,J.H.&Collins,C.E.(2004)Theresurresctionofmultisensorycortexinprimates:
Connectionpatternsthatintegratemodalities,inG.Calvert,C.Spence,&B.E.Stein
(editors),TheHandbookofMultisensoryProcesses,chapter17,pp.285–293(MITPress,
MA).bridge,Cam[140]ofKadunce,Within-andD.C.,VCross-Moaughan,dalitJ.W.,yWSuppreallace,ssioninM.T.,theSupBenedek,eriorG.,etColliculus,al.J(1997)NeurophMecysiolhanisms78
2834–2847.[141]Kailath,T.(1974)Aviewofthreedecadesoflinearfilteringtheory,IEEETInformTheory
146–181.20[142]Kalmijn,A.J.(1988)Hydrodynamicandacousticfielddetection,inJ.Atema,R.R.Fay,A.N.
Popper,&W.N.Tavolga(editors),SensoryBiologyofAquaticAnimals,chapter4,pp.83–130
NY).ork,YNew(Springer,[143]Kandel,E.R.,Schwartz,J.H.,&Jessell,T.M.(2000)Principlesofneuralscience(McGraw-
Hill,NewYork),4th(international)edition.
[144]Kay,S.M.(1993)FundamentalsofStatisticalSignalProcessing(PrenticeHall,UpperSaddle
NJ).er,Riv[145]Kempter,R.,Gerstner,W.,&vanHemmen,J.L.(1999)Hebbianlearningandspikingneurons,
112
Bibliography
PhysRevE594498–4514.
[146]Kempter,R.,Leibold,C.,Wagner,H.,&vanHemmen,J.L.(2001)Formationoftemporal-
featuremapsbyaxonalpropagationofsynapticlearning,ProcNatlAcadSciUSA98
4166–4171.[147]King,A.J.(1999)Sensoryexperienceandtheformationofacomputationalmapofauditory
spaceinthebrain,Bioessays21900–911.
[148]King,A.J.(2009)Visualinfluencesonauditoryspatiallearning,PhilTransRSocB364
331–339.[149]King,A.J.&Hutchings,M.E.(1987)Spatialresponsepropertiesofacousticallyresponsive
neuronsinthesuperiorcolliculusoftheferret:Amapofauditoryspace,JNeurophysiol57
596–624.[150]inKing,theA.J.visual,Hutcandhings,auditoryM.E.,repreMosenore,tationsD.R.,in&theBlakemore,mammalianC.sup(1988)eriorDevcolliculus,elopmentalNatureplasticit332y
73–76.[151]King,A.J.&Palmer,A.R.(1983)Cellsresponsivetofree-fieldauditorystimuliinguinea-pig
superiorcolliculus:Distributionandresponseproperties,JPhysiol342361–381.
[152]King,A.J.&Parsons,C.H.(1999)Improvedauditoryspatialacuityinvisuallydeprived
ferrets,EurJNeurosci11(11)3873–3883.
[153]Knierim,J.J.&vanEssen,D.C.(1992)Neuronalresponsestostatictexturepatternsinarea
V1ofthealertmacaquemonkey,JNeurophysiol67961–980.
[154]Knudsen,E.I.(1982)Auditoryandvisualmapsofspaceintheoptictectumoftheowl,J
1177–1194.2Neurosci[155]Knudsen,E.I.(1999)Mechanismsofexperience-dependentplasticityintheauditorylocaliza-
tionpathwayofthebarnowl,JCompPhysiolA185305–321.
[156]Knudsen,E.I.(2002)Instructedlearningintheauditorylocalizationpathwayofthebarn
owl,Nature417322–328.
[157]Knudsen,E.I.,Blasdel,G.G.,&Konishi,M.(1979)Soundlocalizationbythebarnowl(Tyto
alba)measuredwiththesearchcoiltechnique,JCompPhysiolA1331–11.
[158]Knudsen,E.I.&Brainard,M.S.(1991)Visualinstructionoftheneuralmapofauditoryspace
inthedevelopingoptictectum,Science25385–87.
[159]Knudsen,E.I.&Knudsen,P.F.(1985)Visionguidestheadjustmentofauditorylocalization
inyoungbarnowls,Science230545–548.
[160]Knudsen,E.I.&Knudsen,P.F.(1989)Visioncalibratessoundlocalizationindevelopingbarn
owls,JNeurosci93306–3313.
[161]Knudsen,E.I.,duLac,S.,&Esterly,S.D.(1987)Computationalmapsinthebrain,Annu
41–65.10NeurosciRev[162]Kobayashi,S.,Amemiya,F.,Kishida,R.,Goris,R.C.,etal.(1995)Somatosensoryandvisual
correlationintheoptictectumofapython,Pythonregius:ahorseradishperoxidaseand
golgistudy,NeurosciRes22315–323.
[163]Kobayashi,S.,Kishida,R.,Goris,R.C.,Yoshimoto,M.,etal.(1992)Visualandinfraredinput
tothesamedendriteinthetectumopticumofthepython,Pythonregius:electron-microscopic
350–352.597ResBrainevidence,[164]Koch,C.(1999)BiophysicsofComputation(OxfordUniversityPress,Oxford).
[165]K¨ording,K.&Wolpert,D.M.(2004)Bayesianintegrationinsensorimotorlearning,Nature
244–247.427[166]Krauzlis,R.J.,Basso,M.A.,&Wurtz,R.H.(1997)Sharedmotorerrorformultipleeye
movements,Science2761693–1695.
[167]Krauzlis,R.J.,Liston,D.,&Carello,C.D.(2004)Targetselectionandthesuperiorcolliculus:
Goals,choicesandhypotheses,VisionRes441445–1451.
[168]Krochmal,A.R.,Bakken,G.S.,&LaDuc,T.(2004)Heatinevolutionskitchen:evolutionary
perspectivesonthefunctionsandoriginofthefacialpitofpitvipers(viperidae:Crotalinae),
113
Bibliography
4231–4238.207BiolExpJ[169]Krueger,L.E.(1989)ReconcilingFechnerandStevens:Towardaunifiedpsychophysicallaw,
BehavBrainSci12251–320.
[170]Kuttruff,H.(1991)RoomAcoustics(ElsevierAppliedScience,LondonandNewYork),3rd
edition.[172][171]LamLaming,b,H.D.(1932)(1997)HydrTheoMedynamicsasurement(CamofbridgeSensationUniversit(OxfordyPress),Univ6thersityed.,Oxfedition.ord).
[173]Lebart,K.,Boucher,J.M.,&Denbigh,P.N.(2001)Anewmethodbasedonspectral
subtractionforspeechdereverberation,ActaAcoustica87(3)359–366.
[174]Leibold,C.,Kempter,R.,&vanHemmen,J.L.(2001)Temporalmapformationinthebarn
owlsbrain,PhysRevLett87248101.
[175]Li,Z.(2002)Asaliencymapinprimaryvisualcortex,TrendsCognSci6(1)9–16.
[176]Liman,E.R.(2006)ThermalgatingofTRPionchannels:Foodforthought?,SciStke326.
[177]Linden,D.J.&Connor,J.A.(1995)Long-termsynapticdepression,AnnuRevNeurosci18
319–357.[178]Lingenheil,M.(2004)TheoriederBeuteortungbeimKrallenfrosch,Master’sthesis,Technische
Universit¨atMu¨nchen.
[179]U.Luksch,Windhorst,H.(2008)&M.C.SensorimotorHirschin(editors),tegration:EncyclopOpticediaoftectum,NeurinM.osciencD.eBinder,(Springer,N.NewHirokYawork,a,
NY).[180]Luksch,H.,Gauger,B.,&Wagner,H.(2000)Acandidatepathwayforavisualinstructional
signaltothebarnowl’sauditorysystem,JNeurosci20RC70.
[181]Luo,L.&Flanagan,J.G.(2007)Developmentofcontinuousanddiscreteneuralmaps,Neuron
284–300.56[182]Lynn,G.W.(1931)Thestructureandfunctionofthefacialpitofthepitvipers,AmJAnat
97–133.49[183]Ma,W.J.,Beck,J.M.,Latham,P.E.,&Pouget,A.(2006)Bayesianinferencewithprobabilistic
populationcodes,NatNeurosci91432–1438.
[184]Mach,E.(1866)¨UberdiephysiologischeWirkungr¨aumlichvertheilterLichtreize,Sitzungsber
AkadWissWienII54393–408.
[185]Madison,D.V.,Malenka,R.C.,&Nicoll,R.A.(1991)Mechanismsunderlyinglong-term
potentiationofsynaptictransmission,AnnuRevNeurosci14379–397.
[186]Magosso,E.,Cuppinia,C.,Serino,A.,DiPellegrino,G.,etal.(2008)Atheoreticalstudy
ofmultisensoryintegrationinthesuperiorcolliculusbyaneuralnetworkmodel,Neural
Networks21817–829.
[187]Manley,G.A.,K¨oppl,C.,&Konishi,M.(1988)Aneuralmapofinterauralintensitydifferences
inthebrainstemofthebarnowl,JNeurosci82665–2676.
[188]Markram,H.,L¨ubke,J.,Frotscher,M.,&Sakmann,B.(1997)Regulationofsynapticefficacy
bycoincidenceofpostsynapticAPsandEPSPs,Science275213–215.
[189]McKemy,D.D.,Neuhausser,W.M.,&Julius,D.(2002)Identificationofacoldreceptor
revealsageneralroleforTRPchannelsinthermosensation,Nature41652–58.
[190]McLaughlin,T.&O’Leary,D.D.M(2005)Moleculargradientsanddevelopmentofretinotopic
maps,AnnuRevNeurosci28327–355.
[191]Meszler,R.M.,Auker,C.R.,&Carpenter,D.O(1981)Finestructureandorganizationofthe
infraredreceptorrelay,thelateraldescendingnucleusofthetrigeminalnerveinpitvipers,J
571–584.196NeurolComp[192]Middlebrooks,J.C.&Knudsen,E.I.(1984)Aneuralcodeforauditoryspaceinthecat’s
superiorcolliculus,JNeurosci42621–2634.
[193]Miller,K.(1970)Leastsquaresmethodsforill-posedproblemswithaprescribedbound,
SIAMJMathAnal152–74.
[194]Mills,A.W.(1958)Ontheminimumaudibleangle,JAcoustSocAm30237.
114
Bibliography
[195]Miyoshi,M.&Kaneda,Y.(1988)InverseFilteringofRoomAcoustics,IEEETAcoust
145–152.36(2)heecSp[196]Moiseenkova,V.,Bell,B.,Motamedi,M.,Wozniak,E.,etal.(2003)Wide-bandspectral
tuningofheatreceptorsinthepitorganofthecopperheadsnake(Crotalinae),AmJPhysiol
598–606.284[197]Molenaar,G.J.(1978)Thesensorytrigeminalsystemofasnakeinthepossessionofinfrared
receptors:II.Thecentralprojectionsofthetrigeminalnerve,JCompNeurol179137–152.
[198]&PMolenaar,.S.UlinskiG.J.(1992)(editors),AnatomySensorimotorandphInteysiologygrationof,infvraredolume17sensitivitofyBioloofgysnakofes,theinRC.eptiliaGans,
chapter5,pp.367–453(UniversityofChicagoPress,Chicago,IL).
[199]Molenaar,G.J.&Fizaan-Oostveen,J.L.F.P.(1980)Ascendingprojectionsfromthelateral
descendingandcommonsensorytrigeminalnucleiinpython,JCompNeurol189555–572.
[200]Mooney,R.D.,Klein,B.G.,&Rhoades,R.W.(1987)Effectsofalteredvisualinputupon
thedevelopmentofthevisualsystemandsomatosensoryrepresentationsinthehamster’s
superiorcolliculus,Neurosci20537–555.
[201]Moore,E.H.(1920)Onthereciprocalofthegeneralalgebraicmatrix,BAmMathSoc26
394–395.[202]Morgan,M.L.,DeAngelis,G.C.,&Angelaki,D.E.(2008)Multisensoryintegrationinmacaque
visualcortexdependsoncuereliability,Neuron59662–673.
[203]Murphey,R.K.(1973)Mutualinhibitionandtheorganizationofanon-visualorientationin
Notonecta,JCompPhysiolA8431–40.
[204]Mysore,S.P.&Quartz,S.R.(2005)Modelingstructuralplasticityinthebarnowlauditory
localizationsystemwithaspike-timedependenthebbianlearningrule,ProcIEEEIntJoint
ConfNeuralNetworks52766–2771.
[205]Nakano,M.,Atobe,Y.,Goris,R.C.,Yazama,F.,etal.(2000)Ultrastructureofthecapillary
pericytesandtheexpressionofsmoothmuscleα-actinanddesmininthesnakeinfrared
sensoryorgans,AnatRec260299–307.
[206]Neely,StephenT.&Allen,JontB.(1979)Invertibilityofaroomimpulseresponse,JAcoust
165–9.66(1)AmcSo[207]Nelson,D.A.(1998)Invitededitorialon”penns1948paperrevisited,JApplPhysiol852–3.
[208]Nelson,M.E.&Bower,J.M.(1990)Brainmapsandparallelcomputers,TrendsNeurosci13
408.–403[209]Newman,E.A.,Gruberg,E.R.,&Hartline,P.H.(1980)Theinfraredtrigemino-tectalpathway
intherattlesnakeandinthepython,JCompNeurol191465–477.
[210]Newman,E.A.&Hartline,P.H.(1981)Integrationofvisualandinfraredinformationin
bimodalneuronsoftherattlesnakeoptictectum,Science213789–791.
[211]Newman,E.A.&Hartline,P.H.(1982)Theinfrared“vision”ofsnakes,SciAm246(3)
98–107.[212]Nilius,B.,Talavera,K.,Owsianik,G.,Prenen,J.,etal.(2005)GatingofTRPchannels:a
voltageconnection?,JPhysiol1235–44.
[213]Norwich,K.H.&Wong,W.(1997)Unificationofpsychologicalphenomena:Thecomplete
formofFechner’slaw,PercPsychophys59929–940.
[214]Olsen,J.F.,Knudsen,E.I.,&Esterly,S.D.(1989)Neuralmapsofinterauraltimeandintensity
differencesintheoptictectumofthebarnowl,JNeurosci92591–2605.
[215]Oppenheim,A.V.,Schafer,R.W.,&Stockham,T.G.(1968)Nonlinearfilteringofmultiplied
andconvolvedsignals,ProcIEEE56(8)1264–91.
[216]vanOpstal,A.J.&Munoz,D.P.(2004)Auditory-visualinteractionssubservingprimategaze
orienting,inG.Calvert,C.Spence,&B.E.Stein(editors),TheHandbookofMultisensory
Processes,chapter23,pp.373–393(MIT,Cambridge,MA).
[217]Oˇguzt¨oreli,M.N.&Caelli,T.M.(1985)Aninverseprobleminneuralprocessing,BiolCybern
239–245.53
115
Bibliography
[218]Panton,R.L.(2005)Incompressibleflow(Wiley),3rdedition.
[219]Pneuronsappas,fromT.C.,pitvipMotamedi,erM.,thermosensors,&Christensen,AmJPhB.N.ysiolCell(2004)PhysiolUnique287temp1219–1228.erature-activated
[220]Patton,P.E.,Belkacem-Boussaid,K.,&Anastasio,T.J.(2002)Multimodalityinthesuperior
colliculus:aninformationtheoreticanalysis,CognitiveBrainRes1410–19.
[221]aPecka,neuronalM.,netZahn,workT.Pfor.,soundSaunier-Reblocalizationori,B.,inSivreveke,erbI.,eranettal.environ(2007)ments.,InhibitingJtheNeurosci.inhibition:27(7)
1782–1790.[222]Pennes,H.H.(1948)Analysisoftissueandarterialbloodtemperaturesintherestinghuman
forearm,JApplPhysiol193–122.
[223]Penrose,R.(1955)Ageneralizedinverseformatrices,CambridgePhilosSoc51406–413.
[224]PNeuronserrault,UseT.J.,VDistinctaughan,OpJ.W.,erationalStein,ModesB.E.,in&theWInallace,tegrationM.T.of(2005)MultisensorySuperiorStimColliculusuli,J
2575–2586.93ysiolNeuroph[225]Pickles,J.O.(1988)AnIntroductiontothePhysiologyofhearing(AcademicPress,London),
edition.2nd[226]Planck,M.(1901)Ueberdasgesetzderenergieverteilungimnormalspectrum,AnnPhys309
553–563.[227]Pouget,A.,Dayan,P.,&Zemel,R.S.(2003)Inferenceandcomputationwithpopulation
codes,AnnuRevNeurosci26381–410.
[228]Press,W.H.,Teukolsky,S.A.,Vetterling,W.T.,&Flannery,B.P.(2007)NumericalRecipes:
TheArtofScientificComputing,3rded.(CambridgeUniversity,Cambridge).
[229]Puetter,R.C.,Gosnell,T.R..,&AmosYahil,A.(2005)Digitalimagereconstruction:Deblur-
ringanddenoising,AnnuRevAstronAstrophys43139–194.
[230]Putzar,L.,Goerendt,I.,Lange,K.,R¨osler,F.,etal.(2007)Earlyvisualdeprivationimpairs
multisensoryinteractionsinhumans,NatNeurosci101243–1245.
[231]Rauschecker,J.P.&Kniepert,U.(1994)Auditorylocalizationbehaviourinvisuallydeprived
cats,EurJNeurosci6(1)149–160.
[232]deRivaz,Peter&Kingsbury,Nick(2001)Bayesianimagedeconvolutionanddenoisingusing
complexwavelets,IEEEImageProc2273–276.
[233]R¨oder,B.,R¨osler,F.,&Spence,C.(2004)Earlyvisionimpairstactileperceptionintheblind,
121–124.14BiolCurr[234]Roder,B.,Teder-Salejarvi,W.,Sterr,A.,Rosler,F.,etal.(1999)Improvedauditoryspatial
tuninginblindhumans,Nature400162–166.
[235]Roper,P.,Bressloff,P.C.,&Longtin,A.(2000)Aphasemodeloftemperature-dependent
mammaliancoldreceptor,NeuralComput121067–1093.
[236]Rosenfeld,D.(2002)NewApproachtoGriddingUsingRegularization,MagnResonMed48
193–202.[237]Rowland,B.A.,Quessy,S.,Stanford,T.R.,&Stein,B.E.(2007)Multisensoryintegration
shortensphysiologicalresponselatencies,JNeurosci275879–5884.
[238]Rowland,B.A.,Stanford,T.R.,&Stein,B.E.(2007)Abayesianmodelunifiesmultisensory
spatiallocalizationwiththephysiologicalpropertiesofthesuperiorcolliculus,ExpBrainRes
153–161.180[239]Rowland,B.A.,Stanford,T.R.,&Stein,B.E.(2007)Amodeloftheneuralmechanisms
underlyingmultisensoryintegrationinthesuperiorcolliculus,Perception36(10)1431–1443.
[240]Rucci,M.,Edelman,G.M.,&Wray,J.(1999)Adaptionoforientingbehavior:Fromthebarn
owltoaroboticsystem,IEEETRobot1596–110.
[241]Rucci,M.,Tononi,G.,&Edelman,G.M.(1997)Registrationofneuralmapsthroughvalue-
dependentlearning:Modelingthealignmentofauditoryandvisualmapsinthebarnowl‘s
optictectum,JNeurosci17334–352.
[242]Sarkar,T.K.,Weiner,D.D.,&Jain,V.K.(1981)SomemathematicalConsiderationsin
116
Bibliography
DealingwiththeInverseProblem,IEEETAntennPropag29373–379.
[243]Sch¨afer,K.,Braun,H.A.,&Hensel,H.(1982)Staticanddynamicactivityofcoldreceptors
atvariouscalciumlevels,JNeurophysiol471017–1028.
[244]Sch¨afer,K.,Braun,H.A.,&K¨urten,L.(1988)Analysisofcoldandwarmreceptoractivityin
vampirebatsandmice,PflugArchEurJPhy412188–194.
[245]Schreiner,C.E.&Langner,G.(1988)Periodicitycodingintheinferiorcolliculusofthecat.
II.Topograhicalorganization,JNeurophysiol601823–1840.
[246]Schroeder,D.M.&Loop,M.S.(1976)Trigeminalprojectionsinsnakespossessinginfrared
sensitivity,JCompNeurol1691–14.
[247]Sejnowski,T.J.(1995)Timeforanewneuralcode?,Nature37621–22.
[248]Seung,H.S.&Sompolinsky,H.(1993)Simplemodelsforreadingneuronalpopulationcodes,
ProcNatlAcadSciUSA9010749–10753.
[249]Shams,L.,Kamitani,Y.,&Shimojo,S.(2000)Whatyouseeiswhatyouhear,Nature408
788.[250]Shumway,C.A.(1989)Multipleelectrosensorymapsinthemedullaofweaklyelectricgymno-
tiformfish.I.Physiologicaldifferences,JNeurosci94388–4399.
[251]Sichert,A.B.,Bamler,R.,&vanHemmen,J.L.(2009)Hydrodynamicobjectrecognition:
Whenmultipolescount,PhysRevLett102058104.
[252]Sichert,A.B.,Friedel,P.,&vanHemmen,J.L.(2006)Snake’sperspectiveonheat:Recon-
structionofinputusinganimperfectdetectionsystem,PhysRevLett9768105.
[253]Sichert,A.B.&vanHemmen,J.L.(2010)Howstimulusshapeaffectslateral-lineperception:
analyticalapproachtoanalyzenaturalstimulicharacteristics,BiolCybern102177–180.
[254]Song,S.,Miller,K.D.,&Abbott,L.F.(2000)Competitivehebbianlearningthroughspike-
timing-dependentplasticity,NatNeurosci3919–926.
[255]Stanford,L.R.&Hartline,P.H.(1980)Spatialsharpeningbysecond-ordertrigeminalneurons
incrotalineinfraredsystem,BrainRes185115–123.
[256]Stanford,L.R.,Schroeder,D.M.,&Hartline,P.H.(1981)Theascendingprojectionofthe
nucleusofthelateraldescendingtrigeminaltract:Anucleusintheinfraredsystemofthe
rattlesnake,Crotalusviridis,JCompNeurol201161–173.
[257]Stavenga,D.G.(2002)Reflectionsoncolourfulommatidiaofbutterflyeyes,JExpBiol205
1077–1085.[258]Steenken,R.,Colonius,H.,Diederich,A.,&Rach,S.(2008)Visualandauditoryinteraction
insaccadicreactiontime:Effectsofauditorymaskerlevel,BrainRes1220150–156.
[259]Stein,B.E.&Gaither,N.S.(1981)Sensoryrepresentationsinreptilianoptictectum:Some
comparisonswithmammals,JCompNeurol20269–87.
[260]Stein,B.E.,Jiang,W.,&Stanford,T.R.(2004)Multisensoryintegrationinsingleneuronsof
themidbrain,inG.Calvert,C.Spence,&B.E.Stein(editors),TheHandbookofMultisensory
Processes,chapter15,pp.243–264(MIT,Cambridge,MA).
[261]Stein,B.E.&Meredith,M.A.(1993)TheMergingoftheSenses(MITPress,Cambridge,
MA).[262]Stein,B.E.&Stanford,T.R.(2008)Multisensoryintegration:Currentissuesfromthe
perspectiveofthesingleneuron,NatRevNeurosci9255–266.
[263]Stryer,L.(1986)Cyclicgmpcascadeofvision,AnnualReviewofNeuroscience987–119.
[264]Sullivan,W.E.&Konishi,M.(1986)Neuralmapofinterauralphasedifferenceintheowl’s
brainstem,ProcNatlAcadSciUSA838400–8404.
[265]Takahashi,T.&Konishi,M.(1986)Selectivityforinterauraltimedifferenceintheowl’s
midbrain,JNeurosci63413–3422.
[266]Terashima,S.I.,Goris,R.C.,&Katsuki,Y.(1970)Structureofwarmfiberterminalsinthe
pitmembraneofvipers,JUltraMolStructR31494–506.
[267]Thomson,R.F.(2001)DasGehirn(Spektrum,Akad.Verl.,Heidelberg,Berlin),3rdedition.
[268]Tikhonov,A.N.,Arsenin,V.Y.,&John,F.(1977)SolutionofIll-PosedProblems(V.H.
117
Bibliography
Winston&Sons,Washington,DC),englishtranslationoftheoriginalRussiantext.
[269]forTikhonotheSv,olA.N.,utionofGoncIll-Poseharskyd,PrA.V.,oblemsStepano(Kluv,werV.V.,&AcademicYagola,A.G.Publishers,(1995)DordrecNumerichalt),Methoenglishds
translationoftheoriginalRussiantext.
[270]Truckenbrodt,E.(1999)Fluidmechanik2(Springer),4thedition.
[271]Udin,S.&Fawcett,J.W.(1988)Formationoftopographicmaps,AnnuRevNeurosci11
289–327.[272]Unoki,M.,Sakata,K.,Furukawa,M.,&Akagi,M.(2004)Aspeechdereverberationmethod
basedontheMTFconceptinpowerenveloperestoration,JAcousticalScienceandTechnology
243–254.25(4)[273]Unoki,Masashi,Furukawa,Masakazu,&Sakata,KeigoanfAkagi,Masato(2004)Animproved
methodbasedontheMTFconceptforrestoringthepowerenvelopefromareverberantsignal,
JAcousticalScienceandTechnology25(4)232–42.
[274]theUrsino,supM.,eriorCuppinia,colliculus:C.,aneuralMagosso,netwE.,orkmoSerino,del,JA.,etComputal.(2009)NeurosciMultisensory26(1)55–73.integrationin
[275]Voets,T.,Droogmas,G.,Wissenbach,U.,Janssens,A.,etal.(2004)Theprincipleof
temperature-dependentgatingincold-andheat-sensitiveTRPchannels,Nature430748–
754.[276]vanderWaerden,B.L.(1957)MathematischeStatistik(Springer,Berlin).
[277]Wallace,M.T.,Carriere,B.N.,PerraultJr,T.J.,Vaughan,J.W.,etal.(2006)Thedevelopment
ofcorticalmultisensoryintegration,JNeurosci26(46)11844–11849.
[278]Wallace,M.T.,PerraultJr,T.J.,Hairston,W.D.,&Stein,B.E.(2004)Visualexperienceis
necessaryforthedevelopmentofmultisensoryintegration,JNeurosci249580–9584.
[279]Wallace,M.T.&Stein,B.E.(1997)Developmentofmultisensoryneuronsandmultisensory
integrationincatsuperiorcolliculus,JNeurosci172429–2444.
[280]Wallace,M.T.&Stein,B.E.(2007)Earlyexperiencedetermineshowthesenseswillinteract,
921–926.97ysiolNeurophJ[281]Wandell,B.A.(1995)FoundationsofVision(SinauerAssociates,Sunderland,MA).
[282]Wehr,Michael&Zador,AnthonyM.(2003)Balancedinhibitionunderliestuningandsharpens
spiketiminginauditorycortex,Nature426442–446.
[283]Wickesberg,R.E.&Oertel,D.(1990)Delayed,frequency-specificinhibitioninthecochlear
nucleiofmice:Amechanismformonauralechosuppression,JNeurosci101762–1768.
[284]Willshaw,D.J.&Malsburg,C.(1976)Howpatternedneuralconnectionscanbesetupby
self-organization,ProcRoySocLondB194431–445.
[285]Winkowski,D.E.&Knudsen,E.I.(2006)Top-downgaincontroloftheauditoryspacemap
bygazecontrolcircuitryinthebarnowl,Nature439336–339.
[286]Wissler,E.H.(1998)’penns1948paperrevisited,JApplPhysiol8535–41.
[287]Witten,I.B.,Knudsen,E.I.,&Sompolinsky,H.(2008)AHebbianlearningrulemediates
asymmetricplasticityinaligningsensoryrepresentations,JNeurophysiol1001067–1079.
[288]Wu,Mingyang&Wang,DeLiang(2006)Atwo-stagealgorithmforone-microphonereverberant
speechenhancement,IEEETAudioSpeech14774–84.
[289]Yang,Y.,Chen,J.,Engel,J.,Pandya,S.,etal.(2006)Distanttouchhydrodynamicimaging
withanartificiallateralline,ProcNatlAcadSciUSA10318891–18895.
[290]Yegnanarayana,B.,Avendano,C.,Hermansky,H.,&SatyanarayanaMurthy,P.(1999)
Speechenhancementusinglinearpredictionresidual,SpeechCommun2825–42.
[291]Yegnanarayana,B.&SatyanarayanaMurthy,P.(2000)EnhancementofReverberantSpeech
UsingLPResidual,IEEETSpeechAudioP8267–81.
[292]Yost,W.A.(1991)Auditoryimageperceptionandanalysis:Thebasisforhearing,HearRes
8–18.56(7)[293]Zeil,J.&Hemmi,J.M.(2006)Thevisualecologyoffiddlercrabs,JCompPhysiolA192
1–25.
118
[294]
[295]
Zhang,
L.L.,
Huizong,
W.T.,
Holt,
Bibliography
W.A.,,sHarriC.E.,
et
al.
(1998)
A
critical
wwindo
for
cooperationandcompetitionamongdevelopingretinotectalsynapses,Nature39537–44.
Zittlau,K.E.,Claas,B.,&M¨unz,H.(1988)Horseradishperoxidasestudyoftectalafferents
withlaevisXenopusin
Beh
olEv
32
208–219.
ecialsp
emphasis
on
their
relationship
to
the
lateral-line
system,
Brain
119
Access to the YouScribe library is required to read this work in full.
Discover the services we offer to suit all your requirements!