Forward Modelling of Resistivity Logs using Neural Networks - Liacs
23.11.2013 • Views
Share Embed Flag

Forward Modelling of Resistivity Logs using Neural Networks - Liacs

Forward Modelling of Resistivity Logs using Neural Networks - Liacs

Forward Modelling of Resistivity Logs using Neural Networks - Liacs

SHOW MORE
SHOW LESS
ePAPER READ
DOWNLOAD ePAPER
liacs.nl

You also want an ePaper? Increase the reach of your titles

YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.

START NOW

<strong>Forward</strong><strong>Modelling</strong><strong>of</strong><strong>Resistivity</strong><strong>Logs</strong> <strong>using</strong><strong>Neural</strong><strong>Networks</strong> August31,1995 P.Demmenie

ation<strong>of</strong>rocks.Tothisend,ameasuringdeviceisloweredinawellboreto Abstract<br />

recordaso-calledresistivitylog.Duetoenvironmentaleectsthisresistivitylogdiersfromtheresistivity<strong>of</strong>theformation,thetrueresistivity.To<br />

hydrocarbons(oilandgas)andwatertodeterminethehydrocarbonsatu-<br />

Intheoilindustry,oneusesthedierenceinelectricalconductivitybetween<br />

invertthemeasuredlogtothetrueresistivityoneusesaniterativeforward modellingprocess,involvingthenumericalsolution<strong>of</strong>dierentialequations. Althoughthecurrentmodellingalgorithmshavesignicantlyimprovedin<br />

Theone-waymappingbetweentheearthmodels(trueresistivitymodeland neuralnetworksareveryfastinproducingoutputtocertaininput. siononawellsite.Therefore,wehaveinvestigatedthefeasibility<strong>of</strong><strong>using</strong> neuralnetworkstoperformtheforwardmodellingprocess.Oncetrained, speedcomparingtoafewyearsago,theyarestillnotfastenoughforinver-<br />

net(6750).Thegeneralizationperformance<strong>of</strong>thesenetsisnotsucient number<strong>of</strong>connectionsbetweentheinputlayerandthehiddenlayer<strong>of</strong>the <strong>of</strong>inputsthatisneededtorepresenttheearthmodel(450)andthehigh dard"fullyconnectednet.However,problemsarisefromthehighnumber environmentalconditions)andthetoolresponsecanbelearnedbya\stan-<br />

forthepurpose<strong>of</strong>resistivityloginversion.<br />

<strong>of</strong>fullyconnectednets.Wehavefoundanarchitecturewhichprovestobe number<strong>of</strong>connectionsinthenetby<strong>using</strong>locallyconnectednetsinstead principalcomponentanalysisandthewavelettransform.Thispreprocessing <strong>of</strong>theinputisquitesuccessful.Wehavealsostudiedmethodstoreducethe Wehavestudiedtwopreprocessingmethodstoreducethenumber<strong>of</strong>inputs:<br />

thesesharedweightsthehiddenlayerperformsaconvolution<strong>of</strong>theinput <strong>of</strong>netisbasedonthelocallyconnectednetsandsharedweights.Dueto quiteecientforthismapping:aconvolutional-regressionnet.Thistype layer.Theconvolutionkernel,theset<strong>of</strong>sharedweights,islearnedbythe netitself.Theweightsharingreducesthenumber<strong>of</strong>connectionsinthenet forwardmodelthatisusedatKSEPLnowadays.Theperformance<strong>of</strong>thenet ismeasuredintheaveragerelativeerrorbetweenthenetworkoututandthe andtheseconstraintsimprovethegeneralizationability<strong>of</strong>thenet. Thisconvolutional-regressionnetisapproximately100timesfasterthanthe whentheaveragerelativeerrorliesbelow5%.Theconvolutional-regression forwardmodeloutput.Theneuralnetcanbeusedasafastforwardmodel, netachievesanaccuracy<strong>of</strong>approximately8%onaresistivitylogcoming froma\real"oilwell.Furtherimprovementsinaccuracycanbeachieved by<strong>using</strong>amorerepresentativetrainingset.Evenwithlessaccuracythe neuralnetcouldbeusedasinitialstartfortheforwardmodellingprocess.<br />

i

Preface Thisreportisthemasterthesis<strong>of</strong>PamelaDemmenieforgraduatingfrom theRijksUniversiteitLeiden(RUL)atthedepartment<strong>of</strong>ComputerScience. andDr.IdaSprinkhuizen-Kuyper(RUL). ItdescribesaprojectthatwasperformedattheKoninklijkeShellExploratie enProduktieLaboratorium(KSEPL)inRijswijkfromDecember1994until August1995.TheprojectwassupervisedbyDr.GuozhongAn(KSEPL)<br />

Chapter3and4describetheactualexperimentsfordatawithoutandwith groundonforwardmodellingandashortintroductioninneuralnetworks. Thereportisdividedintosixchapters.Therstchapterprovidesaback-<br />

invasion.Inchapter5wepresenttheresults<strong>of</strong>thetrainedneuralnetworks Thesecondchapterdescribesallthemethodswehaveusedintheexperiments.Thesemethodsinvolveinputreductionsandarchitectureconstraints.<br />

onrealisticloggingdataandinchapter6wesummarizetheconclusions<strong>of</strong> thisproject. AllneuralnetsimulationswererunonSparc10andSparc20(Unix)stationsandshouldeventuallyberunonanIBMR600workstation(theapproximationtimesweremeasuredonthistype<strong>of</strong>workstation).Weused<br />

withothersimulators:Stuttgart<strong>Neural</strong>NetworkSimulator(SNNS)and Aspirin/MIGRAINES.Theadvantagesanddisadvantages<strong>of</strong>thesenetwork theXerionsimulator,versions3.1and4.0.Wehavealsoexperimented<br />

Sprinkhuizen-Kuyperfortheirsupportandideasduringthisproject.Of Acknowledgements First<strong>of</strong>allIwouldliketothankmysupervisorsGuozhongAnandIda simulatorsareoutlinedinappendixA.<br />

vanDijkandLeonHoman,whohaveprovidedthedatathatwehavebeen coursetheprojectwouldnothavebeencompletedwithoutthehelp<strong>of</strong>Niels workingwithandhelpedmeinmyunderstanding<strong>of</strong>theforwardmodelling versityfortheirsupportandcompanyduringtheninemonthsIhavebeen process.IalsoliketothankthestudentsfromKSEPLandfromtheuni-<br />

PimBussinkforword2onpage42. workingontheproject.Andlastbutnotleasteverlastinggratitudegoto<br />

ii

Contents Abstract Preface 1<strong>Forward</strong><strong>Modelling</strong><strong>of</strong><strong>Resistivity</strong><strong>Logs</strong> ii 1i<br />

1.1<strong>Resistivity</strong>Logging::::::::::::::::::::::::1 1.2Inversionby<strong>Forward</strong><strong>Modelling</strong>:::::::::::::::::5 1.2.1Inversion:::::::::::::::::::::::::5 1.1.1Loggingtools:::::::::::::::::::::::1<br />

1.2.2<strong>Forward</strong><strong>Modelling</strong>::::::::::::::::::::6 1.1.2Environmentaleects::::::::::::::::::3<br />

1.3<strong>Neural</strong><strong>Networks</strong>:::::::::::::::::::::::::7 1.3.3Thetrainingset:::::::::::::::::::::10 1.3.1Networkarchitecture:::::::::::::::::::9 1.3.4Generalization::::::::::::::::::::::11 1.3.2TheLearningMethod::::::::::::::::::9<br />

2Inputrepresentationandarchitecturedesign 1.4<strong>Forward</strong><strong>Modelling</strong><strong>using</strong>a<strong>Neural</strong>Network::::::::::12 2.1Inputrepresentation:::::::::::::::::::::::15 1.3.5Localminima:::::::::::::::::::::::12<br />

2.1.1Discretizedslidingwindow::::::::::::::::16 2.1.2Attributes:::::::::::::::::::::::::18 15<br />

2.2Preprocessing:::::::::::::::::::::::::::20 2.1.3Inputandoutputscaling::::::::::::::::19<br />

2.3Architectureconstraints:::::::::::::::::::::27 2.3.1Fullyconnectednets:::::::::::::::::::27 2.2.1PrincipalComponentAnalysis:::::::::::::20<br />

2.3.2Locallyconnectednets::::::::::::::::::28 2.2.2Wavelettransform::::::::::::::::::::22<br />

2.3.6Convolutional-regressionnetwork::::::::::::34 2.3.5TimeDelayednetwork::::::::::::::::::33 2.3.4Convolutionalnetwork::::::::::::::::::32 2.3.3Symmetryconstraints::::::::::::::::::28<br />

3<strong>Forward</strong>modellingwithoutmudinvasion 2.4Errorfunction:::::::::::::::::::::::::::35 3.2Experimentingwiththenetworkarchitecture:::::::::38 3.3Experimentingwiththesize<strong>of</strong>theslidingwindow::::::39 3.1Experimentingwithdierentscalingmethods:::::::::38 37<br />

3.4Summaryandresults:::::::::::::::::::::::39

4<strong>Forward</strong>modellingwithmudinvasion 4.3Experimentingwithinputreductionmethods:::::::::46 4.2Experimentingwithdierentinputrepresentations::::::44 4.1Scaling<strong>of</strong>theparameters::::::::::::::::::::43 4.3.1Usingdierentsamplingmethods::::::::::::46 42<br />

4.4Creatingamorerepresentativetrainingset::::::::::53 4.3.3Reducingtheinputsbyremovingwaveletcoecients:50 4.3.2Reducingtheinputbyprojectiontoprincipalcomponents::::::::::::::::::::::::::::47<br />

4.6Experimentingwitharchitectureconstraints::::::::::57 4.5Intermediateresultsinputrepresentations:::::::::::55<br />

4.7Intermediateresultsarchitecturedesign::::::::::::65 4.6.4Experimentingwithconvolutionalregressionnets:::59 4.6.3Usingsymmetryconstraints:::::::::::::::57 4.6.2Experimentingwithlocallyconnectednets:::::::57 4.6.1Experimentingwithfullyconnectednets::::::::57<br />

5Theneuralnetworkasfastforwardmodel 5.2Applicationtoearthmodelswithinvasion:::::::::::73 5.1Applicationtoearthmodelswithoutinvasion:::::::::68 4.8Summaryandresults:::::::::::::::::::::::65<br />

5.3Applicationtorealisticearthmodel:::::::::::::::73<br />

6Conclusions 6.2Methods::::::::::::::::::::::::::::::83 6.1<strong>Neural</strong>networkasfastforwardmodel?:::::::::::::81 6.2.1Inputrepresentation:::::::::::::::::::83 6.2.2Inputreduction::::::::::::::::::::::84<br />

A<strong>Neural</strong>networksimulators 6.3Application<strong>of</strong>theconvolutional-regressionnet::::::::85 6.2.3Architecturedesign::::::::::::::::::::85I

List<strong>of</strong>Tables<br />

54231 Parametersforearthmodelswithinvasion.::::::::::42 Performanceonearthmodelswithoutinvasion(1).::::::40 Parametersforearthmodelswithoutinvasion.::::::::37 Averagerelativeerrorforearthmodelswithoutinvasion(1).39 Thetoolresponse.::::::::::::::::::::::::13<br />

876 plingperiods.:::::::::::::::::::::::::::45 Comparinguniformsamplingmethodswithdierentsam-<br />

Comparingdiscretizedslidingwindowagainstattributesrepresentation.::::::::::::::::::::::::::::45<br />

9 Comparinguniformandnon-uniformsamplingmethods.::45 Comparinguniformsampledinputwithoutandwithprojectiontoprincipalcomponents.::::::::::::::::::48<br />

10Comparingnon-uniformsampledinputwithoutandwithprojectiontoprincipalcomponents.::::::::::::::::49<br />

11Comparinginputrepresentationswithdierentnumber<strong>of</strong> 12Comparinginputrepresentationswithdierentnumber<strong>of</strong> 13Comparingtrainingset<strong>of</strong>targetlogsandtrainingset<strong>of</strong> waveletcoecients(1).::::::::::::::::::::::51 waveletcoecients(2).::::::::::::::::::::::51<br />

16Comparingdierentlocallyconnectednets.::::::::::58 15Comparingdierentfullyconnectednets.:::::::::::58 14Comparingtrainingset<strong>of</strong>targetlogsandtrainingset<strong>of</strong>dif- cultparts<strong>of</strong>targetlogs.::::::::::::::::::::54 coarsersampledtargetlogs.:::::::::::::::::::54<br />

19Comparing\wavelet"netwithoutandwithsymmetryconstraints.::::::::::::::::::::::::::::::6tryconstraints.::::::::::::::::::::::::::60<br />

constraints.::::::::::::::::::::::::::::58 18Comparinglocallyconnectednetswithoutandwithsymme-<br />

17Comparingfullyconnectednetswithoutandwithsymmetry<br />

20Resultsforconvolutional-regressionnets(1).:::::::::63 21Resultsforconvolutional-regressionnets(2).:::::::::63 22Averagerelativeerrorandperformanceforearthmodelswith 23Averagerelativeerrorandperformanceforearthmodelswith 24Averagerelativeerrorandperformanceforearthmodelswith invasion(Shallow-log).::::::::::::::::::::::65 andwithoutinvasion(Shallow-log).:::::::::::::::66 andwithoutinvasion(Deep-log).::::::::::::::::67

List<strong>of</strong>Figures 132 Laboratoryresistivitymeasuringapparatuswithunguarded planarelectrodes.:::::::::::::::::::::::::1<br />

part<strong>of</strong>theformation.::::::::::::::::::::::3 InputtotheDeepLaterolog(left)andtheShallowLaterolog (right).TheDeepLaterologgetsinformationfromalarger electrodesinstead<strong>of</strong>planarelectrodesandtheDual-Laterolog.2 Modication<strong>of</strong>thelaboratoryapparatus<strong>using</strong>cylindrical<br />

456 Environmentaleects:(A)idealsituation,(B)dippingformation,(C)cavedboreholes,(D)deviatedboreholesand<br />

Theinversionprocessandforwardmodelling<strong>of</strong>resistivitylogs.6 (E)horizontalboreholes.::::::::::::::::::::4<br />

10Part<strong>of</strong>amodelwithinputsignalRtandoutputsignalsDeeplog(LLd)andShallow-log(LLs).TheinputsignalsRxoand<br />

879 Localminimaintheerrorsurface.:::::::::::::::12 Adiagram<strong>of</strong>neuronj.::::::::::::::::::::::8 Aformationanditsmodel.:::::::::::::::::::13 Transferimpedances.:::::::::::::::::::::::6<br />

12Whentheslidingwindowissmallerthanthelargestbedinthe 11Example<strong>of</strong>slidingwindowinputrepresentation.Asliding dxoareomitted.:::::::::::::::::::::::::14<br />

situation,althoughthetargetdiersalot.:::::::::::17 model,theinputtothenetwillbethesameforthesketched theinputmodels.:::::::::::::::::::::::::15 window<strong>of</strong>sizewisplacedaroundthepoint<strong>of</strong>interestalong<br />

16Theinputsignaliswrittenasaweightedcombination<strong>of</strong>rectangularblockfunctions.Thecoecientsciareusedasinputs<br />

totheneuralnet.:::::::::::::::::::::::::25<br />

13Standardnormaldistribution.::::::::::::::::::19 15HaarwaveletfromboxfunctionWHaar(x)=(2x)(2x1).23 14Most<strong>of</strong>thevariation<strong>of</strong>thedataliesinthedirection<strong>of</strong>1.:21<br />

17Boundariesoutsidethecenter<strong>of</strong>thewindowarenotdescribed 20Adiscretizedinputsignalanditsmirrorimage.::::::::29 21Symmetryconstraintsonfullyconnectednet.:::::::::30 18Fullyconnected(left)andlocallyconnected(right)neuralnets.28 19Receptiveeldswhichoverlap13.::::::::::::::::29 accurately.:::::::::::::::::::::::::::::26<br />

23Symmetryconstraintsonwaveletnets.Thecoecientsare 22Symmetryconstraintsonlocallyconnectednet.Thereceptiveeldsareconstrainedsymmetrically(receptiveeldfis<br />

24Convolutionalnetwork.::::::::::::::::::::::32 constrainedperdetaillevel.::::::::::::::::::31 constrainedtoeldFf+1forFelds).::::::::::30

29Response<strong>of</strong>theneuralnettothedicultmodelshownabove.41 28ModelthatcausesdicultiesinapproximatingtheDeep-log.41 27Thenetworkandrelativeerrorforxedproportionsd=a.::35 26Convolutional-regressionnetwork.::::::::::::::::34 25Timedelayedneuralnetwork.::::::::::::::::::33<br />

31Non-uniformsamplinginslidingwindow.:::::::::::47 30Smalldierenceininputwithattributesinputrepresentation.44<br />

33Loss<strong>of</strong>informationpervariableforTest7.Originalinput 32Loss<strong>of</strong>informationpervariableforTest6.Originalinput consists<strong>of</strong>364inputs,comingfromanon-uniformsampled slidingwindow.::::::::::::::::::::::::::48 slidingwindow.::::::::::::::::::::::::::49 consists<strong>of</strong>3128inputs,comingfromauniformsampled<br />

35Intermediateresults<strong>of</strong>trainingthreedierentneuralnetson 34Notalltransitionsaredetectedbywaveletcoecients.:::52<br />

37Convolutional-regressionnet.Thersthiddenlayerconsists 36Convolutional-regressionnet.Thefeaturemapsintherst 6models.:::::::::::::::::::::::::::::56<br />

<strong>of</strong>threesets<strong>of</strong>featuremaps.Eachsetconsists<strong>of</strong>threemaps andisconnectedtoone<strong>of</strong>thevariablesintheinputlayer.:61 layer.:::::::::::::::::::::::::::::::59 hiddenlayerareconnectedtoallthevariablesintheinput<br />

39Activation<strong>of</strong>rsthiddenlayerperfeaturemapforTest21. 38Trueresistivityprole<strong>of</strong>modelAatdepth615feet.:::::64<br />

41Worstcaseneuralnetapproximation<strong>of</strong>Deep-log.Average 40Intermediateresults(2)<strong>of</strong>trainingdierentneuralnetson6 models.::::::::::::::::::::::::::::::66 TheseactivationsareformodelAatdepth615feet.This<br />

relativeerroris14.2%.:::::::::::::::::::::69 layerconsists<strong>of</strong>6featuremaps<strong>of</strong>27nodeseach.:::::::64<br />

42Bestcaseneuralnetapproximation<strong>of</strong>Deep-log.Average 43Worstcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average 44Bestcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris2.6%.::::::::::::::::::::::70<br />

45Examples<strong>of</strong>earthmodelsusedforthenetInvasion.:::::75 46Worstcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris8.5%.::::::::::::::::::::::71<br />

47Bestcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris1.2%.::::::::::::::::::::::72<br />

49<strong>Neural</strong>netapproximation<strong>of</strong>Deep-log.Averagerelativeerror 48A(realistic)earthmodel.::::::::::::::::::::78 relativeerroris7.5%.::::::::::::::::::::::76 relativeerroris4.5%.::::::::::::::::::::::77 is7.6%.:::::::::::::::::::::::::::::79

50<strong>Neural</strong>netapproximation<strong>of</strong>Shallow-log.Averagerelative erroris8.3%.:::::::::::::::::::::::::::80

1<strong>Forward</strong><strong>Modelling</strong><strong>of</strong><strong>Resistivity</strong><strong>Logs</strong><br />

andhydrocarbonsdonotconductelectriccurrents,whereastheformation Therockporescanbelledwithwaterorhydrocarbons(oilandgas).Rock 1.1<strong>Resistivity</strong>Logging<br />

hydrocarbonsaturationwithintherocks.Thehydrocarbonsaturationisan waterdoes.Onereasonformeasuringtheresistivityistodeterminethe Aformationconsists<strong>of</strong>severallayers(beds)<strong>of</strong>rock,whichcontainpores.<br />

indicationforthepresence<strong>of</strong>oil.Asimpliedexpression<strong>of</strong>thisquantitative aspectisexempliedbyArchie'sequation<br />

HereRwistheresistivity<strong>of</strong>thewaterintherockpores,Fistheformation Sw=FnRw<br />

factorgenerallyassumedtobederivablefromaknowledge<strong>of</strong>therockresistivity,Swisthewatersaturation(percentage<strong>of</strong>porespaceoccupiedby<br />

water),Rtisthemeasuredrockresistivityandnisanempiricallydeter-<br />

(Moran1985). 1.1.1Loggingtools minedsaturationexponent.HydrocarbonsaturationShisequalto1Sw<br />

Rt (1)<br />

Most<strong>of</strong>thephysicsbehindtheresistivityloggingtechniquescanbefound in(Moran1985).Inthissectionwewilldiscussthebasicideasbehindthese techniques.<br />

1<br />

electrode B<br />

V<br />

electrodes. Asheet<strong>of</strong>materialwhoseresistivityistobedeterminedisplacedbetween Figure1:Laboratoryresistivitymeasuringapparatuswithunguardedplanar andincontactwithelectrodesAandB,both<strong>of</strong>areaSasisshownin<br />

area S<br />

A<br />

distance dr in meter

electrode A<br />

I<br />

0<br />

I<br />

1<br />

I<br />

2<br />

I<br />

1<br />

2 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />

(m)yields Figure1.AvoltageVisappliedtotheelectrodes,resultinginacurrent application<strong>of</strong>Ohm'slawformaterialthicknessdr(m)andresistivity I.IfIisdistributeduniformlyovertheareaSandiszerooutside,an<br />

Theresistivityisfoundbymeasuringthevoltagedropbetweentheelectrodes.Inpractice,thecurrentIwillnotbeuniformlydistributedoverthe<br />

discs,butwilltendtobemaximumattheedgesandminimuminthecentre. V=ISdr (2)<br />

disc<strong>of</strong>areaS0carryingacurrentI0surroundedbytheremainder<strong>of</strong>the Toimproveonthisscheme,onesplitsthediscsothatonehasasmallcentral discsheldatthesamepotentialVasS0.Thiswillresultinnearlyconstant currentdensityoverS0.ThemeasuredcurrentI0willbe\focused".The resistivityisnowgivenmoreaccuratelythanbeforeby =S0 drV I0 (3)<br />

A1<br />

I<br />

0<br />

I<br />

0<br />

A0<br />

I<br />

Figure2:Modication<strong>of</strong>thelaboratoryapparatus<strong>using</strong>cylindricalelectrodesinstead<strong>of</strong>planarelectrodesandtheDual-Laterolog.<br />

D D0<br />

1<br />

A2<br />

I0<br />

I1<br />

I<br />

2<br />

I<br />

Thisideaisthebasicingredient<strong>of</strong>theLaterolog.TherealLaterologuses<br />

1<br />

cylindricalelectrodesinstead<strong>of</strong>planarelectrodesasindicatedinFigure2<br />

I<br />

2<br />

B<br />

aDeepLaterolog.TheDeepLaterologhasitscurrentreturnelectrodeB (left).TheDualLaterologconsists<strong>of</strong>aShallow(Pseudo)Laterologand<br />

SHALLOW DEEP<br />

remotelyatthesurface,whichresultsincurrentsasshowninFigure3.The

1.1<strong>Resistivity</strong>Logging 3<br />

DeepLaterologreadsfarintotheformation.TheShallowLaterologhas itscurrentreturnelectrodeplacedaboveandbelowtheelectrodeA,which makesthecurrentsbendbacktothetoolasshowninFigure3.ThisLaterologreadstheformationclosetothetool,whichmakesitmoresensitive<br />

totheinvadedzone(seeSection1.1.2). Bothmeasurementsaremadesimultaneouslyby<strong>using</strong>dierentfrequencies; quencyfortheDeepLaterolog. arelativelyhighfrequencyfortheShallow-Laterologandaverylowfre-<br />

part <strong>of</strong> the<br />

formation that<br />

is seen by the<br />

tool<br />

part <strong>of</strong> the<br />

formation that<br />

is seen by the<br />

tool<br />

formation. Figure3:InputtotheDeepLaterolog(left)andtheShallowLaterolog<br />

tool<br />

tool<br />

1.1.2Environmentaleects (right).TheDeepLaterologgetsinformationfromalargerpart<strong>of</strong>the<br />

Severalanomalies,likeinvasion,dippingbeds,washed-outboreholesand<br />

in(Gianzero1977),(Asquith1982),(Chemali,Gianzero&Strickland1983) occurareshowninFigure4andmoredetaileddescriptionscanbefound <strong>of</strong>theformation.Theseanomalieshaveaneectonthetoolresponse, whicharediculttomodel.Situationsinwhichtheseenviromentaleects veryresistiveformations,areencounteredwhenmeasuringtheresistivity<br />

pressure,whichpreventsblow-outs. and(Chemali,Gianzero&Su1988). Wellsaredrilledwithrotarybits.Specialdrillingmudisusedascirculating uid.Themudremovescuttingsfromthewellbore,lubricatesandcoolsthe drillbitandhelpsmaintaininganexcess<strong>of</strong>boreholepressureoverformation

4 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />

Thispressuredierenceforcessome<strong>of</strong>thedrillinguidtoinvadeporous andpermeableformation.Inthisprocess<strong>of</strong>invasionsolidparticles(clay mineralsfromthedrillingmud)aretrappedontheside<strong>of</strong>theboreholeand formaso-calledmudcake.Thepart<strong>of</strong>theformationwhichisinvadedby mudltrateiscalledtheinvadedzone.<br />

A<br />

B<br />

tool<br />

current flow<br />

C D E<br />

Thesize<strong>of</strong>theboreholeandthemudcakeresistivityinuencethemeasured (C)cavedboreholes,(D)deviatedboreholesand(E)horizontalboreholes. Figure4:Environmentaleects:(A)idealsituation,(B)dippingformation,<br />

eectiveinmoderatelysalinetoverysalinemud,andlesseectiveinfresh resistivity.Thesmoothing<strong>of</strong>anomaliesonthelogbytheboreholeisquite IndippingformationstheDualLaterolog-curvesvaryslowlyacrossbed portionsandtheerrorduetoshoulderbedeect(theinuence<strong>of</strong>thebeds mud.Thesize<strong>of</strong>theboreholeandtheresistivity<strong>of</strong>thedrillinguidare<br />

adjacenttothecurrentbed)isdierentfromthenon-dippingcase. boundaries.Theapparentbedthicknessisincreasedinpredictablepro-<br />

takenxedinourstudy.<br />

tweenbedsalsoaectthetoolreadings.Formationslikelimestoneand occuronlyatabruptchangesinholediameter. Veryresistiveformationsandformationswithhighresistivitycontrastsbe-<br />

<strong>of</strong>theincreasedholediameter.Anomalousreadings<strong>of</strong>theDeepLaterolog Incavedboreholes(wash-out)theShallowLaterologissensitivetotheeect<br />

dolomitecanhaveresistivitiesover2000m,whilemostformationshavea resistivitybetween1and70m. OnlytheShallowLaterologexhibitssomesensitivitytoeccenteringinlarge boreholes. Thetoolresponseisessentiallyimmunetoasmalleccentering<strong>of</strong>thetool.

1.2Inversionby<strong>Forward</strong><strong>Modelling</strong> 5<br />

Theoveralleect<strong>of</strong>anellipticalboreholeisthatitproducescharacteristicresponseswhichliebetweenthoseobtainedintwocircularholeswith<br />

diametersequaltothemajorandminoraxes<strong>of</strong>theellipticalhole. importantinconductivemudthanitisinnon-conductivemud.TheShallowLaterologhasmuchlessshoulderbedeectthantheDeepLaterolog,<br />

Deviatedboreholesandhorizontalboreholesalsogivedierenttoolreadings.Comparethecurrentowintheidealcase(Figure4)andthesetypes<br />

<strong>of</strong>boreholes. 1.2Inversionby<strong>Forward</strong><strong>Modelling</strong> 1.2.1Inversion Thefoundresistivitylogs,Shallow-andDeep-log,havetobeinvertedto thetrueresistivity<strong>of</strong>theformation.Thisisdonebyaniterativeforward rstcorrectedwithchartbookcorrections(tornadochartsforexample;for boreholesize,invasion,etc.)Thenaninitialguessfortheformationmodelis modellingprocess,whichissketchedinFigure5.Theactualeldlogsare<br />

TheshoulderbedcorrectionrequiredfortheDeepLaterologismuchless especiallyforbedthicknessesabovetenfeet.<br />

boreholediameterandbeddingdip,thicknessandresistivity(Anderson& geometryand\parametervalues"|numbersassignedtovariablessuchas Barber1990).Thenthetoolphysicsisusedtocomputeanexpectedlog, made.Thistrialmodelincludesadescription<strong>of</strong>theboreholeandformation whichiscomparedwiththeactualeldlog.Ifthematchisnotgoodenough,<br />

<strong>Forward</strong><strong>Modelling</strong>)andamatchingprocedure.Thelatterprocedureimplies guessedformationmodeltotheexpectedDeep-andShallow-log(so-called Theprocessconsists<strong>of</strong>twosteps:thefunctionapproximationfromthe isiterateduntilthetwologsmatchsatisfactorily. theinitialtrialmodelisalteredandthecalculationrepeated.Thisprocess<br />

aminimalization<strong>of</strong>amatchingerrorbetweenthetwologs.Thiscanbe maximumentropymethods(Anderson&Barber1990). donebyhandor<strong>using</strong>sophisticatedalgorithmsliketheleast-squaresand

6 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />

Formation model<br />

Expected log<br />

<strong>Forward</strong><br />

<strong>Modelling</strong><br />

Adapt<br />

Matching<br />

initial guess<br />

<strong>of</strong> formation<br />

procedure<br />

1.2.2<strong>Forward</strong><strong>Modelling</strong> Figure5:Theinversionprocessandforwardmodelling<strong>of</strong>resistivitylogs.<br />

<strong>of</strong>theprocess.TocomputetheLaterologresponseforagivenelectrodearray Theforwardmodellingpart<strong>of</strong>theinversionisthemosttime-consumingpart<br />

Actual measured field V1 I1<br />

Z 12<br />

Z<br />

11<br />

Z<br />

V<br />

01<br />

Z<br />

0 I<br />

00<br />

0<br />

Z<br />

02<br />

anddistribution<strong>of</strong>resistivities,itissucienttodeterminetheassociated<br />

Z<br />

transferimpedances.ThetransferimpedanceZijisequaltoVj<br />

V2<br />

I<br />

22<br />

2<br />

thevoltagemeasuredatpartj<strong>of</strong>theelectrodecongurationandIiisthe Figure6:Transferimpedances.<br />

currentemittedbyparti.Aspecicelectrodecongurationanditstransfer Ii,whereVjis<br />

ELECTRODE<br />

TRANSFER<br />

CONFIGURATION IMPEDANCES

1.3<strong>Neural</strong><strong>Networks</strong> 7<br />

insimpliedmodels(concentriccylindricalboundariesorplaneboundaries) impedancesareshowninFigure6.Forthecomputation<strong>of</strong>theresponse onecanuseanalyticalapproaches,butwithextendedelectrodesforexample,<br />

certainboundaryconditions. Zijinvolvesthesolution<strong>of</strong>Laplace'sequationintwospacevariablesunder beenpublished(Moran1985).Thedetermination<strong>of</strong>thetransferimpedances dippinglayers,theproblemis<strong>of</strong>suchcomplexitythatnoresultshaveasyet theproblemcanonlybehandledbynumericalmethods.Inthecase<strong>of</strong><br />

Thereareanumber<strong>of</strong>methodsthatcanbeusedtosolvetheseboundary valueproblems,wementiontheFiniteElementMethod,theBoundaryelementornite-dierencetechniqueandaHybridMethod(Gianzero,Lin&<br />

Su1985).<br />

(horizontal)dependenceistreatednumericallyandtheaxial(vertical)dependenceanalytically.Itcombinesthemodeconceptinwave-guidetheory<br />

withtheFiniteElementMethod.Thismethod,employedbyGianzero,has ElementMethod(Gianzeroetal.1985). beenabletosimulatea100footlogwith25bedsinlessthantwelveminutesonanIBM3081,whichisapproximately8timesfasterthantheFinite<br />

directconsequence<strong>of</strong>minimizingthetotalenergy<strong>of</strong>thesystem. ciple(Chemalietal.1983).ItcanbeshownthatLaplace'sequationisa TheHybridMethodisaseparation<strong>of</strong>variablesapproachwheretheradial TheFiniteElementMethodisanumericalmethodbasedonanenergyprin-<br />

A<strong>Neural</strong>Networkconsists<strong>of</strong>anumber<strong>of</strong>layersthatconsist<strong>of</strong>nodes(neurons).Aneuronreceivesinputfromtheneuronsitisconnectedtoandcan<br />

initsturnserveasinputtootherneurons.Aconnectionbetweennodei<strong>of</strong> Agoodintroductioncanbefoundin(Haykin1994). Inthissectionwewillonlydescribethebasicingredients<strong>of</strong>neuralnetworks. 1.3<strong>Neural</strong><strong>Networks</strong><br />

wji.Thetotalinputujfornodejis acertainlayerandanodej<strong>of</strong>anotherlayerischaracterizedbyaweight<br />

wherexiistheactivation<strong>of</strong>aninputnodeiandthesummationrunsover allnnodesthatnodejreceivesinputfrom. uj=nXi=1wjixi (4)

8 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />

.<br />

.<br />

.<br />

.<br />

.<br />

.<br />

.<br />

.<br />

.<br />

.<br />

activation function<br />

summing<br />

function<br />

input<br />

signals<br />

synaptic<br />

weights<br />

treshold<br />

output<br />

θ<br />

φ(.)<br />

x<br />

x<br />

x<br />

w<br />

w<br />

w<br />

1<br />

2<br />

n<br />

j<br />

j<br />

1<br />

2<br />

n<br />

y<br />

u<br />

j<br />

j<br />

j<br />

j<br />

Figure7:Adiagram<strong>of</strong>neuronj.<br />

Nowthisnodeusesanactivationfunctiontodetermineitsownactivation<br />

andoutput yj=(ujj) (5)<br />

Hereisacertainactivationfunction,whichisusuallyasigmoidfunction<br />

forthehiddennodesandalinearfunctionfortheoutputnodeinregressionproblems,andjisacertainthresholdfornodej.Figure7showsa<br />

visualization<strong>of</strong>thisprocess.<br />

Thisprocedure<strong>of</strong>determiningtheinputandtheoutput<strong>of</strong>anodeisdonefor<br />

allnodes,excepttheinputnodes.Theinputnodesreceivetheiractivation<br />

directlyfromoutside(fromaleforexample).Inthiscasetheinputissome<br />

kind<strong>of</strong>representation<strong>of</strong>theformationmodel.<br />

Thetimethatisneededtodeterminetheoutput<strong>of</strong>theneuralnetfora<br />

certaininputcanbeexpressedinthenumber<strong>of</strong>connections(weights)<strong>of</strong>the<br />

net.Theoperationsdonebythenetareweightmultiplications,summations<br />

andcalculatingtheactivation.<br />

Determiningtheoutput<strong>of</strong>thenetwillnottakemuchtime.However,the<br />

process<strong>of</strong>makingthenetlearntheproblem,training,takesconsiderably<br />

moretime.When<strong>using</strong>neuralnetworks,thefollowingaspectsareimportant<br />

forthetrainingtimeandthegeneralizationability.Anetissaidtogeneralize<br />

wellifitproduces(nearly)correctoutputforinputthatwasnotusedduring<br />

itstraining.<br />

Thearchitecture<strong>of</strong>thenet.Largenets(highnumber<strong>of</strong>weights)learn<br />

slowlyandusuallydonotgeneralizewell.Thisisduetothefactthat<br />

theneuralnet\remembers"itstrainingexamplesifithastoomuch<br />

freedom(toomanyweights).

1.3<strong>Neural</strong><strong>Networks</strong> 9<br />

Thetrainingmethod.Severaltrainingmethodsforsupervisedlearningexist,forexampleback-propagation,conjugategradient,steepesingmeansthatthenetproducessomeoutput(theactualresponse)<br />

certainparametersisveryimportantanddicult.Supervisedlearn-<br />

andcorrectsitsbehaviouraccordingtothecorrectoutput(desiredre-<br />

descent,momentumdescent.Forsome<strong>of</strong>thesemethodsthetuning<strong>of</strong> sponse).Formultilayerfeedforwardnetworks(networksthathavecon-<br />

nectionsdirectedfromtheinputtotheoutput),themostwidelyused algorithmistheback-propagationalgorithm.Therearetwophases intheBP-learning.Intherstphase,theforwardphase,theinputsignalspropagatethroughthenetworklayerbylayer,eventually<br />

responsesoproducediscomparedtothedesiredresponse.Theerror signalsgeneratedbythiscomparisonarethenpropagatedbackwards producingsomeresponseattheoutput<strong>of</strong>thenetwork.Theactual<br />

Thetrainingset.Thetrainingsetshouldberepresentativeforthe throughthenetinthesecond,thebackwardphase.Moreinformation<br />

problem,otherwisethenetisnotabletolearntheproblemortogeneralizewell.Withconjugategradienteachtrainingexampleisevaluated<br />

ontrainingmethodscanbefoundin(Haykin1994).<br />

duringtraining(batchtraining),soalargetrainingsetcausesalong<br />

1.3.1Networkarchitecture inSection1.3.3. trainingtime.Moreonthecreation<strong>of</strong>thetrainingsetcanbefound<br />

perlayerandtheconnectionsbetweenthelayers;itdescribeswhatthenet Thenetworkarchitectureisadescription<strong>of</strong>thelayers,thenumber<strong>of</strong>nodes nodes<strong>of</strong>onelayerreceiveinputfromallnodesinthepreviouslayer.This lookslike.Forexampleafullyconnectedneuralnetisanetinwhichthe<br />

cationsforthelearningandgeneralizationability<strong>of</strong>thenet. Anothermethodisalocallyconnectednetwork,whichisdescribedlater Thearchitecturedeterminesthenumber<strong>of</strong>weightsandhascertainimpli-<br />

isthemostcommonarchitecture,butitleadstoalargenumber<strong>of</strong>weights. whenweturntoreceptiveeldsandconvolutionalnetworks.<br />

1.3.2TheLearningMethod Anetusesalearningmethodtominimalizetheerrorinthenet.Thetraining setconsists<strong>of</strong>Ppatterns(xp;dp).Here,xpistheinputpatternpanddp isthedesiredoutputforthisinputpattern.Foracertaininputxpthe networkcalculatesanoutputap.Theerrorisgivenbythequadraticsum

10 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />

<strong>of</strong>thedierence<strong>of</strong>thisoutputandthedesiredoutputdp:<br />

isthattheweightsareadaptedduringtrainingsothatEisminimalized. Here,Pisthenumber<strong>of</strong>examples.Theideabehindthelearningmethods E=PXp=1(dpap)2 (6)<br />

adaptedinthedirectionwiththesteepestdescent Thisisdonebyadownhilltechnique,gradientdescent.Theweightsare 0B@Weight<br />

wji1CA=0B@learning-rate parameter 1CA0B@local Theweightcorrectiondependsonalearningparameter,thelocalgradient gradient1CA0B@inputsignal <strong>of</strong>neuronj toneuroni 1 CA(7) andtheinputsignal<strong>of</strong>neuronitoneuronj.IfEnolongerdecreasesthe<br />

betweenthedirectionandthegradientvectors.TheCGmethodisthemost bythegradientdescentmethod,byincorporatinganintricaterelationship <strong>of</strong>thenormalgradientdescentmethod.Itavoidsthezigzagpathfollowed trainingstops.<br />

convenientalgorithm,becauseitneedsnotuningparametersanditisfaster Wehaveusedtheconjugategradient(CG)method,whichisanadaptation<br />

thannormalback-propagationifthesize<strong>of</strong>thetrainingsetisnottoolarge. WehaveusedtheXerion(version3.1and4.0)simulatortotrainthenetworks (formoredetailsonneuralnetworksimulatorsseeAppendixA).<br />

layer.Howthisisdone,isdiscussedinthenextchapter.Nowwewillfocus Theinputhastobepresentedtothenet,sothephysicalmodelmustbe transformedintoaset<strong>of</strong>numbersthatfunctionasactivations<strong>of</strong>theinput 1.3.3Thetrainingset<br />

onthescaling.<br />

theweightswillingeneralbefarapartaswell.Thiswillslowdownthe alsohavealargevariation.Soiftheinputsdonotlieinasmallrange, inputshaveaveryhighratio(forexamplei1=1andi2=1000)theweights range[1;1].Aswehaveseenthenetcalculatesaweightedinput.Iftwo Theinputtothenetmustbescaledsothevaluesliemoreorlessinthe<br />

set<strong>of</strong>weights.Anotherpoint<strong>of</strong>viewisthatlargeinputvalueswillhave training,becausealargerweightspacehastobesearchedforanoptimum moreinuenceontheactivation<strong>of</strong>thenodestheyareconnectedto.Inthis waywehavealreadybuildsomepriorknowledgeintothenet.Toavoidthis, wescalealltheinputstoasmallrange.Whencertaininputsareimportant, inSection2.1.3. thenetcanlearnthatitself.Moreonthissubject<strong>of</strong>scalingcanbefound

1.3<strong>Neural</strong><strong>Networks</strong> 11<br />

Atrainingsetshouldbesucientlylarge.Althoughthereisnogeneral prescription<strong>of</strong>howlargeatrainingsetshouldbe,therearesome\rules", likethefollowingfromBaumandHassler(Haykin1994).Anetworkwill followingtwoconditionsaremet: almostcertainlyprovidegeneralization(seenextsection),providedthatthe<br />

2.Thenumber<strong>of</strong>examples,Pusedintrainingis 1.Thefraction<strong>of</strong>errorsmadeonthetrainingsetislessthat2.<br />

Here,Wisthetotalnumber<strong>of</strong>weightsandMisthetotalnumber P32Wln32M<br />

<strong>of</strong>hiddennodes.Thisformulaprovidesadistribution-free,worstcaseestimationforthesize<strong>of</strong>thetrainingsetforasingle-layerneural<br />

(8)<br />

Atrainingsetshouldalsoberepresentative.Thismeansthattheexamples inthetrainingsetarerandomlygeneratedanddistributedoverthewhole inputspace. networkthatissucientforagoodgeneralization.<br />

1.3.4Generalization<br />

bythreefactors: isdeterminedbyitsabilitytopredictoutputstoinputsithasnotseen duringtraining,whichiscalledgeneralization.Generalizationisinuenced Althoughaneuralnetcanlearnanyinput-outputmapping,itsapplicability<br />

thesizeandrepresentativeness<strong>of</strong>thetrainingset,<br />

Withto<strong>of</strong>ewexamplesthenetjustmemorizesthetrainingsetandexhibits thephysicalcomplexity<strong>of</strong>theproblemathand. thearchitecture<strong>of</strong>thenetwork,<br />

poorgeneralization.Ifthenumber<strong>of</strong>examplesismorethanthenumber <strong>of</strong>weights,thenetwillgeneralizebetter.Widrow'srule<strong>of</strong>thumb(Haykin setsize<strong>of</strong>approximately10timesthenumber<strong>of</strong>weightswhentheerroron thetrainingsetis10%. Inourproject,wewillcomparedierentarchitecturesononespecictrainingset.Thearchitecturethatgivesthebestresults,interms<strong>of</strong>training<br />

error,generalization(testing)errorandcomplexity(number<strong>of</strong>weights),will<br />

1994)comesfromequation8andstatesthatinpracticeweneedatraining<br />

betrainedonalargertrainingset.

12 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />

global minimum<br />

Error surface<br />

local minimum<br />

local minimum<br />

InFigure8(left)theerrorsurface,belongingtospecicweightvectors, 1.3.5Localminima Figure8:Localminimaintheerrorsurface.<br />

global minimum<br />

local minimum<br />

initialization<br />

technique,whichmaycausethealgorithmtogettrappedinalocalminimum isshown.Thelearningalgorithmthatweuseisbasicallyahill-descending<br />

minimum<br />

errorsmallerthanthepreviousone.Butsomewhereelseintheweightspace thereexistsanotherset<strong>of</strong>synapticweightsforwhichthecostfunctionis intheerrorsurface,althoughweareinterestedintheglobalminimum.The algorithmgetstrapped,becauseitcannotndadirectionwhichmakesthe<br />

tializations.ThisprocessisshowninFigure8(right);withdierentweight smallerthanthelocalminimuminwhichthenetworkisstuck.Onemethod toavoidlocalminimaisretrainingtheneuralnetwithdierentweightini-<br />

initializationthenetwillconvergetodierentminima.<br />

andtheShallow-log.Theformationisdescribedbyanumber<strong>of</strong>beds.For eachbedanumber<strong>of</strong>radialzonesaregivenandforeachradialzonethe 1.4<strong>Forward</strong><strong>Modelling</strong><strong>using</strong>a<strong>Neural</strong>Network Thegoal<strong>of</strong>thisprojectistoobtainagoodapproximation<strong>of</strong>theDeep-<br />

resistivity(m)anditssize(inch)isgiven.Therstradialzoneisthebore<br />

Thetoolresponseconsists<strong>of</strong>twocontinuouslogs(theShallow-logandthe thetrueresistivity.Allthiscanbedescribedwithamodellikeshownin holewithitsradiusandtheresistivity<strong>of</strong>thedrillinguid,thenextradial<br />

Deep-log)likeshowninFigure10andTable1.ThemodelshowninFigure Figure9(thecorrespondingformationisshownontherightside). zonedescribestheinvasion(ifthereisany)andthelastradialzonedescribes<br />

innity(thisisthesameforallmodels,indicatinganinniteshoulderbed). 9contains80beds.Therstbedhas3radialzonesanditstartsatminus<br />

aresistivity(Rt)<strong>of</strong>27m.Thelastradialzonehasaradius<strong>of</strong>innity 45inchandaresistivity(Rxo)<strong>of</strong>1.90m.Andnallythethirdzonehas andthemudresistivityis0.05m.Thesecondzonehasaradius(dxo)<strong>of</strong> Therstradialzone<strong>of</strong>thisbed(theborehole)hasaradius<strong>of</strong>4.25inch

1.4<strong>Forward</strong><strong>Modelling</strong><strong>using</strong>a<strong>Neural</strong>Network 13<br />

number <strong>of</strong> beds<br />

borehole radius<br />

4.25 inch<br />

(resistivity drilling fluid<br />

0.05 Ωm)<br />

1e+09<br />

Rt = 27 Ωm<br />

80<br />

3<br />

-1e+09<br />

one bed<br />

dxo = 45 inch<br />

0.05<br />

4.25<br />

1.90<br />

27.00<br />

45.00<br />

1e+09<br />

Rxo = 1.90 Ω m<br />

start depth <strong>of</strong><br />

bed<br />

number <strong>of</strong><br />

3 0.00<br />

Rt = 12 Ωm<br />

radial<br />

borehole<br />

zones<br />

0.05 4.25 radius<br />

0.90 19.00<br />

dxo = 19 inch<br />

invasion<br />

resistivity<br />

radius<br />

12.00 1e+09<br />

<strong>of</strong> drilling<br />

fluid<br />

(otherwisetherewouldbeanotherradialzone).<br />

radius <strong>of</strong><br />

Theneuralnetcanbeusedasafastforwardmodelwhentheaveragerelative Figure9:Aformationanditsmodel.<br />

virgin errorbetweentheforwardmodeloutputdandtheneuralnetworkoutputa<br />

invasion<br />

true<br />

Rxo = 0.90 Ω m<br />

liesbelow5%.<br />

resistivity<br />

resistivity<br />

depth 0.0.110610E+02.330318E+01 0.2.108760E+02.325301E+01 0.4.106962E+02.320733E+01 Table1:Thetoolresponse.<br />

0.6.105092E+02.316235E+01 LLd LLs<br />

radiusandresistivity<strong>of</strong>thedrillinguidarexedto4.25inchand0.05m Theearthmodelsinthisprojectarecreatedbyassigningrandomnumbers withinacertainrangefortheparametersRt,Rxoanddxo.Theborehole . . .<br />

forwardmodelthatisusedatKSEPL. respectively.Thetoolresponsestothesemodelsarecalculatedwiththe

14 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />

Ohm meter<br />

True resistivity Rt<br />

Deep-log<br />

Shallow-log<br />

(LLd)andShallow-log(LLs).TheinputsignalsRxoanddxoareomitted. Figure10:Part<strong>of</strong>amodelwithinputsignalRtandoutputsignalsDeep-log<br />

100<br />

10<br />

5400 5420 5440 5460 5480 5500<br />

depth (feet)

15<br />

point <strong>of</strong> interest<br />

(window is centered around this point)<br />

Tool response<br />

True resistivity Rt<br />

Invasion resistivity Rxo<br />

AsdescribedinSection1.3,thetrainingtimemainlydependsonthesize<strong>of</strong> increasethegeneralizationcapability,weliketokeepthenetassmallas thenetandthesize<strong>of</strong>thetrainingset.Todecreasethetrainingtimeand 2Inputrepresentationandarchitecturedesign<br />

possible.Thiscanbeachievedbychoosingacompactinputrepresentation andbydecreasingthenumber<strong>of</strong>freevariables(weights)inthenet.The inputrepresentationhastobecompactandappropriatetotheproblem. T<strong>of</strong>acilitatethelearningprocess,weemploytwomethods.Firstlywepreprocesstheinputandsecondlyweforcecertainconstraintsonthenetwork<br />

methodcontributest<strong>of</strong>acilitatinglearningonlyiftheresultingstructure networkwhilemakingonlyasmallcomputationaloverhead.Thesecond reectsthedesigner'saprioriknowledge<strong>of</strong>theproblem.Otherwisethe aterepresentation<strong>of</strong>theinputwhichsimpliestheproblemfortheneural architecture.Thepurpose<strong>of</strong>therstmethodistocreateanintermedi-<br />

2.1Inputrepresentation networkisaprioribiasedtowardswrongsolutions.<br />

Wearelookingforacompactandappropriateinputrepresentation.Given anearthmodel,describedbyn(2+r2)+1values(fornbeds<strong>of</strong>rradial<br />

Figure11:Example<strong>of</strong>slidingwindowinputrepresentation.Aslidingwindow<strong>of</strong>sizewisplacedaroundthepoint<strong>of</strong>interestalongtheinputmodels.<br />

Invasion diameter dxo<br />

w

16 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />

(thetargetisalsocalledthedesiredoutput).Foroneformationwehavea toolresponse<strong>of</strong>mfeet,sampledeverytfeet.Thisproducesmt+1targets Thetoolresponseataspecicdepthisusedasthetarget<strong>of</strong>theneuralnet perlog.Butwhatdoweuseasinputtoproducethistarget?Weassume zones),thenetshouldbeabletoproducethetoolresponseatanydepth.<br />

slidingwindow.TheslidingwindowapproachisshowninFigure11. forthetoolresponseatthatdepth.Thispart<strong>of</strong>theformationiscalleda thatapart<strong>of</strong>theformation,centeredaroundacertaindepth,isresponsible<br />

Therstmethodisquitestraightforward.Itsamplesthepart<strong>of</strong>theformationthatliesinthewindow,without<strong>using</strong>knowledgeabouttheinput<br />

orrelationsbetweeninputs. Thesecondmethodlooksmoreliketheoriginalmodeldescriptionanduses thefactthattheformationisdescribedbybeds.Inthemodeleachbedis centeredaroundthepoint<strong>of</strong>interestbyanumber<strong>of</strong>features. describedbyanumber<strong>of</strong>values,whichcouldbeseenasattributes<strong>of</strong>that<br />

modelintheslidingwindowandbydescribingthebedsthatlieinthe Theslidingwindowisdescribedintwoways:bydiscretizingtheformation<br />

2.1.1Discretizedslidingwindow bed.Intheattributesapproachwedescribethebedsthatlieinawindow,<br />

Whenthereisinvasion,seeSection1.1.2,theformationmodelcontainsthree<br />

inputswhenthereisnoinvasionand3timesthisnumberwhenthereis sampledwithinthiswindowwithasamplingperiods,resultinginws+1 isplacedalongtheinputlogsaroundthepoint<strong>of</strong>interest.Themodelsare containsonlythevariableRt.Weuseaslidingwindow<strong>of</strong>xedsizew,which variablesRt,Rxoanddxo.Whenthereisnoinvasiontheformationmodel<br />

<strong>of</strong>theslidingwindow: invasion. Thefollowingaspectsshouldbetakenintoaccountindeterminingthesize Thesize<strong>of</strong>thetool.Thecurrentsowingfromthetoolpenetrate theformation.Thecurrents<strong>of</strong>theShallowLaterologpenetratethe formationandreturntothetopandbottom<strong>of</strong>thetool.Thepart<strong>of</strong><br />

Thetype<strong>of</strong>targetlog(DeeporShallow).AsshowninFigure3,the largeasthetoolitself.Thetoolisapproximately30ft,whichgives theformationthatthetoolreceivesinformationfrom,isatleastas<br />

DeepLaterologreceivesinformationfromalargerpart<strong>of</strong>theformation.Thecurrents<strong>of</strong>theShallowLaterologreturntothetoolitself<br />

anindicationthatthewindowsizeshouldalsobeatleast30ft.<br />

windowsizefortheDeep-logshouldbelargerthanfortheShallow-log anddonotpenetratetheformationmuch.Thisindicatesthatthe

Ω meter<br />

60<br />

45<br />

30<br />

input model Rt<br />

15<br />

2.1Inputrepresentation 17<br />

Thesize<strong>of</strong>thebeds.Whathappensifthewindowissmallerthan thelargestbedintheformationisshowninFigure12.Thesliding windowislocatedmorethanonceinthesamebed,producingthe sameinput,butpossiblynotthesametarget.Nowtheneuralnet hastolearnf(x)=y1andf(x)=y2,makingthetheproblemnondeterministic.Conictingexamplesmakeitverydicultforthenet<br />

dierenttargetsforoneinput. tolearntheproblem,becauseitadaptsitsweightstoreproducetwo<br />

0<br />

second (first<br />

model,theinputtothenetwillbethesameforthesketchedsituation, Figure12:Whentheslidingwindowissmallerthanthelargestbedinthe<br />

shifted by some samples)<br />

to net looks like<br />

30 30 30 30 .... 30 30 30 30<br />

WeexpectthenetthatistrainedontheShallow-logwillperformbetter, becauseitneedsasmallerwindowthantheDeep-log.Thesamplingperiod althoughthetargetdiersalot.<br />

first window, to looks like<br />

30 30 30 30 ... 30 30 30 30<br />

<strong>of</strong>thetargetlogdeterminesthenumber<strong>of</strong>examplesweobtainperlog.In reasons,wehavetakenveloggingpointsperfeet.Thesamplingperiodin theslidingwindowisimportantfortheresolution<strong>of</strong>thebedboundaries. realapplicationsonetakestwologgingpointsperfeet,butforeciency<br />

windowasforthetargetlog.Inthiscasethatwouldmean<strong>using</strong>asampling withsfeetresolution.Whenwewanttohaveatleastthesameaccuracy period<strong>of</strong>0.2feet. Whenweuseasamplingperiod<strong>of</strong>sfeet,wecandescribeabedboundary<br />

Thisinputrepresentationisnotverycompact.Whenweuseaslidingwindow<strong>of</strong>25.4feetandasamplingperiod<strong>of</strong>0.2feet,wehave384inputs(128<br />

asthetargetlog,weshouldusethesamesamplingperiodforthesliding<br />

pervariableand3variableswhenthereisinvasion).

18 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />

bexedforeachsample.Weuseaxedsizewindowanddescribethebeds 2.1.2Attributes Theinputmodeldescribesanumber<strong>of</strong>bedboundaries,eachdescribedby anumber<strong>of</strong>radialzones.Theneuralnetrequiresthenumber<strong>of</strong>inputsto thatoccurwithinthiswindow.Ifthewindowcontainslessthanthexed shoulderbeds.Theorderinwhichthebedsarepresentedtotheneuralnet number<strong>of</strong>beds,weadd\default"beds.Thesebedsfunctionasinnite<br />

thisbedarepresentedandthebedsadjacenttothosebedsandsoon. shouldbexed(inourcaseitispresentedrst).Thenthebedsadjacentto Eachbedisdescribedbyanumber<strong>of</strong>attributes(alsocalledfeatures).The thetargetsignalforexample(probablythebedinthecenter<strong>of</strong>thewindow) isimportant.Thelocationintheinput<strong>of</strong>thebedthathasmostinuenceon<br />

contrastbetweentwobedsisdenedasv1=v2andthedierenceasv1v2 forvaluesv1andv2.Theattributesweuseare: 1.thetrueresistivity<strong>of</strong>thebed(Rt);<br />

4.theinversedistancetotheloggingpoint(pointsclosetothebedboundaryareconsideredtobemoreimportantthanpointsthatliefurther<br />

away); 3.theinvasionradius<strong>of</strong>thebed(dxo); 2.theinvasionresistivity<strong>of</strong>thebed(Rxo);<br />

5.thecontrastbetweenthetrueresistivity<strong>of</strong>thisbedandthebedthat 6.thecontrastbetweentheinvasionresistivity<strong>of</strong>thisbedandthebed<br />

7.thecontrastbetweentheinvasionradius<strong>of</strong>thisbedandthebedthat thatliesbelowthisbed;<br />

8.thedierencebetweenthetrueresistivity<strong>of</strong>thisbedandthebedthat<br />

9.thedierencebetweentheinvasionresistivity<strong>of</strong>thisbedandthebed<br />

10.thedierencebetweentheinvasionradius<strong>of</strong>thisbedandthebedthat thatliesbelowthisbed;<br />

centbeds.Theinversedistancetotheloggingpointis0forthedefaultbeds Thedefaultbedshavenocontrast(1)andnodierence(0)withtheiradja-<br />

(thebedscontinuetoinnity).Ifwedescribenbedsinthechosenwindow, liesbelowthisbed.<br />

thisresultsin10ninputs.

2.1Inputrepresentation 19<br />

Thisapproachismorecompactthanthediscretizedslidingwindowapproach,butitisdiculttochooseappropriatefeaturesthatwillfacilitate<br />

problem,whenthesefeaturesarenotdescribingtheproblemwell. befacilitated.Itcouldevenmakeitdicultfortheneuralnettolearnthe thelearningprocess.Weassumethecontrastsanddierencesareimportant anddescribetheproblemwell.Ifthisisnotthecase,thelearningwillnot<br />

combination<strong>of</strong>logarithmicandnormalizationscalingandfortheothervariablesanormalizationscaling.Thetrueresistivity(andthetoolresponse)<br />

Asdiscussedintherstchapter,theinputshouldbeapproximatelyscaledto thedomain[1;+1].Forthetrueresistivityandthetoolresponseweusea 2.1.3Inputandoutputscaling<br />

canrangebetween1and2000m.Toreducethisrange,weusealogarithmicscaling.Therange<strong>of</strong>theothervariables,theinvasionresistivity<br />

logarithmicscalingonthesevariables.Thescalingstakethefollowingform andtheinvasionradius,aremuchsmallerandthereforewedonotusethe<br />

where,xistheoriginalinputoroutputvalue,themeanandthestandard x0norm=x x0log=ln(x) (10) (9)<br />

deviation<strong>of</strong>thevariablex.Thefactorisappliedtomakethescaledrange evensmaller(intheexperiments=2). AnormalizedGaussiandistributionisshowninFigure13.Fromthisgure wendthatfor=1,68.27%<strong>of</strong>thevalues(<strong>of</strong>thisspecicdistribution) liebetween-1and+1.Whenweuse=2wendthat95.45%<strong>of</strong>the valuesliebetween-1and+1.<br />

-3 -2 -1 1 2 3<br />

applied. Intheattributedescriptionapproachonlyanormalizedscalingmethodis Figure13:Standardnormaldistribution.<br />

68.27 %<br />

95.45 %<br />

99.73 %

20 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />

quencefortheminimalization<strong>of</strong>thenetworkerror.Formoredetailsonthe Thiscombinedscalingmethod<strong>of</strong>thetoolresponsehasanattractiveconse-<br />

2.2Preprocessing minimalization<strong>of</strong>thenetworkerrorseeSection2.4.<br />

Inthissectionwewilldescribethepreprocessingmethodswehaveusedin theproject.Thepurpose<strong>of</strong>preprocessingistondanintermediateinput representationthatfacilitatesthelearningprocess.Anotheradvantage<strong>of</strong><br />

Rt,Rxoanddxoandsecondlyasthreefunctions<strong>of</strong>thedepthx,Rt(x), windowapproach. Wecanviewtheinputintwoways.FirstlyasthreeN-dimensionalvectors wasonlyappliedtotheinputthatwascreatedbythediscretizedsliding preprocessingisthatwecanreducethenumber<strong>of</strong>inputs.Thepreprocessing<br />

Rxo(x)anddxo(x).Thelatterisactuallyalsoavector,becausethefunctionsareequallysampledwithinaninterval[1;N].Intherstcasewe<br />

projecttheinputtoanM-dimensionalsubspacespannedbytheprinciple components<strong>of</strong>theinput.Intheothercaseweuseaset<strong>of</strong>orthogonalbasis functions,theHaarwavelets,toprojecttheinput. Thedisadvantage<strong>of</strong>inputreductionistheloss<strong>of</strong>information.Hopefully, theinformationthatislosthasanegligibleinuenceonthelearningand generalization<strong>of</strong>thenet. 2.2.1PrincipalComponentAnalysis Thefollowingdescription<strong>of</strong>theprincipalcomponentanalysisistakenfrom (Hertz,Krogh&Palmer1991).<br />

ProjectingthedatafromtheiroriginalN-dimensionalspaceontotheMdimensionalsubspacespannedbythesevectorsperformsadimensionality<br />

reductionthatretainsmost<strong>of</strong>theintrinsicinformationinthedata. inthesubspaceperpendiculartotherst.Withinthatsubspaceitistaken maximumvariance.Thesecondprincipalcomponentisconstrainedtolie alongthedirectionwiththemaximumvariance.Thenthethirdprincipal Therstprincipalcomponentistakentobealongthedirectionwiththe componentistakeninthemaximumvariancedirectioninthesubspace alonganeigenvectordirectionbelongingtothekthlargesteigenvalue<strong>of</strong>the Ingeneralitcanbeshownthatthekthprincipalcomponentdirectionis<br />

analysis(PCA),alsoknownastheKarhunen-Loevetransformincommunicationtheory.Theaimistondaset<strong>of</strong>Morthogonalbasisvectors<br />

Acommonmethodfromstatisticsforanalyzingdataisprincipalcomponent (eigenvectors)thataccountforasmuchaspossible<strong>of</strong>thedata'svariance.<br />

perpendiculartothersttwo,andsoon.AnexampleisshowninFigure14.

2.2Preprocessing 21<br />

ε<br />

1<br />

e1<br />

ε<br />

2<br />

e2<br />

covariancematrix.ThismatrixiscalculatedforPpatternsby Figure14:Most<strong>of</strong>thevariation<strong>of</strong>thedataliesinthedirection<strong>of</strong>1.<br />

Covariance(i;j)=Pp(xpixi)(xpjxj)<br />

e3<br />

Here,xpiisinputi<strong>of</strong>patternp,xpjisinputj<strong>of</strong>patternpandxiand xjarethemeans<strong>of</strong>inputiandinputjrespectively.Thenthematrixis diagonalizedandtheeigenvaluesarecalculated. Theoriginalinputvectorx(Rt;Rxoordxo)iswrittenas wheree1,...,eNaretheoriginalbasisvectorsasshowninFigure14.We canalsowritetheinputvectorinanotherorthogonalset<strong>of</strong>basisvectors,the eigenvectors1;:::;Nx=x1e1+x2e2+:::+xNeN (12)<br />

Toreducethesize<strong>of</strong>thevectorsfromNtoM,weprojectthisvectorto x=(1x)1+(2x)2+:::+(Mx)M+:::+(Nx)N x0=(1x)1+(2x)2+:::+(Mx)M (13)<br />

theneuralnet. Instead<strong>of</strong><strong>using</strong>Nvaluesx1;x2;:::;xNfromtheoriginalinputvectorweuse Mvalues1x;2x;:::;Mx<strong>of</strong>theprojectedinputvectorasinputsfor (14)<br />

P (11)

22 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />

Tocalculatethepercentage<strong>of</strong>informationthatislostbythisprojection,we rsthavetowritex0intheeibasisvectors: Thevaluesaiarecalculatedby x0=a1e1+a2e2+:::+aNeN (15)<br />

patternsintheinputle)by Theinformationthatislostbythisprojectioncanbecalculated(forP ai=NXn=1xnMXm=1mnmi (16)<br />

Loss<strong>of</strong>information=1PPXp=1jjx0pxpjj<br />

2.2.2Wavelettransform jjxpjj (17)<br />

onalbasisfunctionsfi(x) Wecanwriteanyfunctionf(x)asaweightedcombination<strong>of</strong>otherorthog-<br />

f(x)isacontinuousfunction,weusethecoecientsci.Wechoosefunctions Instead<strong>of</strong><strong>using</strong>thevalues<strong>of</strong>f(x)(asinputs),whichmaybeinnitewhen f(x)=Xicifi(x) (18)<br />

fi(x)withproperties,thatmakeiteasiertomanipulatewiththosefunctions AcommonlyknownmethodistheFouriertransform,wheretheorthogonal thanwiththeoriginalfunctionf(x). basisfunctionsaresin(ax)andcos(ax).Thesebasisfunctionsallowyouto describethefunctionondierentfrequencylevels.<br />

anotherinterestingset<strong>of</strong>orthogonalbasisfunctions,calledwavelets.Weare OurinputsignalsRt(x),Rxo(x)anddxo(x)haveaveryspecialshape:they<br />

especiallyinterestedinthesimplestwavelets,theso-calledHaarwavelets. inthiscase,becausewewouldneedaninnitenumber<strong>of</strong>coecientsand basisfunctionstocorrectlymodelthediscretetransitions.Thereishowever areallrectangular\functions".TheFouriertransformisnotappropriate<br />

part<strong>of</strong>thewindow(varyinginsizeandlocation).Thepropertytheydescribeisthedierencebetweentheaveragevalueovertherstandseconestingaboutwaveletsisthattheinputcanbedescribedondierentdetail<br />

levels.Allcoecients,excepttherst,describeaspecicpropertyovera ThesewaveletsareblockfunctionsasshowninFigure16.Whatissointer-<br />

half<strong>of</strong>theirpart<strong>of</strong>thewindow.Therstcoecientdescribestheaverage overthewholewindow.

2.2Preprocessing 23<br />

Wewillrstgiveashortintroductiononwaveletsandhowthecoecients AwaveletisdenedbyW(x)=Xk(1)kc1k(2xk)<br />

arecalculated(takenfrom(Strang1989))andthenwewillexplainwhythe waveletsaresousefulinourproject.<br />

by Here,kistakensymmetricallyaroundzero.Thescalingfunctionisdened (19)<br />

underconditions (x)=Xck(2xk) Zdx=1 (20)<br />

TheHaarwaveletisthesimplestwavelet.Forthiswaveletwechoosec0=1 Xkck=2 (21)<br />

andc1=1.Thescalingfunctionforthesecoecientsisablockfunction denedby (22)<br />

ThewaveletasshowninFigure15,isdescribedby (x)=(10x1<br />

WHaar(x)=(2x)(2x1) 0otherwise (23)<br />

Westartwithavector(f1;f2;:::;fN),whichareN=2jequallysampled (24)<br />

Figure15:HaarwaveletfromboxfunctionWHaar(x)=(2x)(2x1). values<strong>of</strong>afunctionf(x)onaunitinterval.Thisvectorwillbeapproximatedbythesum<strong>of</strong>dierentweightedblockfunctions.InFigure16the<br />

0 1<br />

rst8blockfunctions,belongingtothelevels0,1and2,aregiven.The projectedinputvectorisdescribedbythecoecientsthatcorrespondtothe

24 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />

weighing<strong>of</strong>theseblockfunctions. Howarethesecoecientscalculated?Onavectorx<strong>of</strong>2jvaluesweperform L(x),calculatesthemeanandthesecond,H(x),calculatesthedierence. twooperationsL:R2j!R2j1andH:R2j!R2j1.Therstoperation,<br />

L(x)=k0 B @x1+x2 xN1+xN x3+x4 . 1 CA H(x)=k0 B @x1x2 x3x4<br />

Here,usuallyk=12indecompositionandk=1inreconstruction,but xN1xN . 1 CA (25)<br />

wecouldalsousek=12p2inboththedecompositionandreconstruction, doneintheexperimentsdescribedlater). ThevectorsproducedbyL(x)andH(x)arebothhalfthesize<strong>of</strong>theoriginal whichhastheadvantage<strong>of</strong>normalizingthewaveletsateveryscale(thisis<br />

level,levelj1.Tondthecoecientsatthenextdetaillevel,weperform vector.ThecoecientsfoundbyH(x)arethecoecientsonthenestdetail atthepreviouslevel.Thiscontinuesuntilwereachlevel0. H(x)andthecoecientatlevel0foundbyL(x).So,ifwenameLithe averageoperatoratleveliandHithedierenceoperatoratleveli,thenew theoperationsL(x)andH(x)recursivelyonthevectorproducedbyL(x) Theprojectedinputconsists<strong>of</strong>thecoecientsperdetaillevelfoundby Onlevelithereare2icoecientsproducedbyHi,thedierenceoperator. inputvectorisconstructedby(L0;H0;H1;:::;Hj1). Onlevel0wehaveanextracoecientcomingfromL0.Thetotalnumber <strong>of</strong>inputsis whichisequaltothesize<strong>of</strong>theoriginalinputvector. Wewillnowpresentanexampleforavector<strong>of</strong>length23,x=(1;3;2;2;5;3;8;0). N=1+20+21+:::+2j1=2j (26)<br />

Intherststeponlevelj=2,wegetfork=12 L(x)=0B@24 1 CAandH(x)=0B@1<br />

Wecontinueonlevelj=1withL(x)=x0asinputvector 014 1 CA (27)<br />

L(x0)= 24!andH(x0)= 0! (28)

2.2Preprocessing 25<br />

=<br />

C<br />

C<br />

0<br />

1<br />

x<br />

x<br />

+<br />

+<br />

C<br />

2<br />

x<br />

+<br />

C<br />

3<br />

x<br />

+<br />

C<br />

4<br />

x<br />

+<br />

C x<br />

+<br />

5<br />

C x<br />

+<br />

net.<br />

6<br />

Andnallyonlevelj=0wendforL(x0)=x00 Figure16:Theinputsignaliswrittenasaweightedcombination<strong>of</strong>rectangularblockfunctions.Thecoecientsciareusedasinputstotheneural<br />

C<br />

7<br />

L(x00)=3andH(x00)=1<br />

x<br />

Thevector<strong>of</strong>waveletcoecients,f0(x)is(L(x00);H(x00);H(x0);H(x)) (29)<br />

f(x)= 0 B@ 1 325380 1 CAandf0(x)= 0 B@ 1 3 014 1 CA (30)

26 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />

thewindowismoreimportantthantheinformationontheedges.Weknow exactlywhichcoecientsarecorrespondingtotheedgesandwhichtothe Whatissointerestingaboutthewaveletcoecients,isthattheydescribea center<strong>of</strong>thewindowandwecanremove(some<strong>of</strong>)thecoecientsthatare verylocalarea<strong>of</strong>theinput.Weexpectthattheinformationinthecenter<strong>of</strong><br />

coecient,thetransitionsintherstquarter<strong>of</strong>thewindowcanonlybe describedbysteps<strong>of</strong>w=4<strong>of</strong>coecientc2.Inthiscasewelooseresolution Coecientc4fromFigure16,forexample,describesthetransitionsinthe correspondingtotheedges,whenwearenotinterestedinthisinformation.<br />

Ifwehaveaninputcomingfromaslidingwindow<strong>of</strong>w=25.4feet,sampled ontherstquarter<strong>of</strong>thewindow. rst14<strong>of</strong>thewindow(sizew)withsteps<strong>of</strong>sizew=8.Ifweremovethis<br />

every0.2feet,wehave128inputvaluesand7detaillevels0to6.Onthe nerdetaillevels(5and6)weremoveanumber<strong>of</strong>coecientsontheedges aredescribedbysteps<strong>of</strong>0.2feetandontheedgesbysteps<strong>of</strong>0.8feet(this isdetaillevel4). InFigure17isshownwhathappenswhenanumber<strong>of</strong>coecientsonthe accuracyastheoriginalinputinthecenter.Inthecenterthetransitions <strong>of</strong>thewindow.Welooseresolutionattheedges,butweretainthesame<br />

edgesisremoved.Thetransitionsinthecenteraredescribedveryaccurately, butontheedgesweuselargerstepsthanintheoriginalinput(atransition isthenapproximatedbystepsinstead<strong>of</strong>onetransition).<br />

center <strong>of</strong> sliding window<br />

accurately. Figure17:Boundariesoutsidethecenter<strong>of</strong>thewindowarenotdescribed<br />

approximation<br />

true boundaries

2.3Architectureconstraints 27<br />

tailedinformationinthecenterispreserved.If,forexample,weremove20 5,weremoved7.8feetonlevel6andalso7.8feetatlevel5.Thismeans thatattheseparts<strong>of</strong>thewindowwehavearesolution<strong>of</strong>0.8feetandinthe Thecoecientsareremovedfromtheedges<strong>of</strong>thewindow,soallthede-<br />

coecientsoneachsideonlevel6and10coecientsoneachsideonlevel center9.8feetwehavearesolution<strong>of</strong>0.2feet. Howmuchinformationislostbythisoperationdependsonthelocation <strong>of</strong>thebedboundariesandthesharpness<strong>of</strong>thetransitions.Aresistivity contrastisapproximatedbysmallstepsinstead<strong>of</strong>onetransition.Ahard duringtrainingandtesting. imationaectsthegeneralizationperformanceinanegativewaywillshow measurefortheloss<strong>of</strong>informationisdiculttogive.Whethertheapprox-<br />

2.3Architectureconstraints Thesecondmethodtoreducethecomplexity<strong>of</strong>thenet,aswehadalready onthearchitecture.Theadvantage<strong>of</strong>thismethodisthatnoinformation mentionedinthebeginning<strong>of</strong>thissection,isforcingcertainconstraints islost,likeinthepreprocessingmethods.Wecanalsoreectourprior knowledgeabouttheinputinthedesign<strong>of</strong>thenet.Thisfacilitatesthe By<strong>using</strong>locallyconnectedneurons(receptiveelds),thenetcontainsmuch learningprocessandhopefullywillimprovethegeneralization<strong>of</strong>thenet.<br />

architectures. advantagesanddisadvantages<strong>of</strong>variousfullyandlocallyconnectednetwork isfurtherinvestigatedinSection2.3.4.Inthissectionwewilldiscussthe thenumber<strong>of</strong>weights,byforcingsome<strong>of</strong>theweightstobeequal.This lessweightsthanthefullyconnectednets.Wecanevenfurtherreduce<br />

leftside<strong>of</strong>Figure18(inthisgureonlytheconnectionstothersthidden 2.3.1Fullyconnectednets nodearedrawn).Allnodes<strong>of</strong>onelayerareconnectedtoallnodes<strong>of</strong>the Acommonnetworkarchitectureisafullyconnectednetasshownonthe<br />

muchfreedomanditwillnotgeneralize.Morehiddenlayerscanbeadded previouslayer.Thenumber<strong>of</strong>nodes<strong>of</strong>thehiddenlayerisveryimportant. theproblem.Butitshouldalsonotbetoolarge,otherwisethenethastoo toimproveonthetrainingandgeneralizationresults.Usuallyonehidden Thisnumbershouldnotbetoosmall,otherwisethenetisnotabletolearn<br />

<strong>of</strong>theinput. layerisenoughtolearnaproblem,butsometimesanextralayerhelpsto combinethefeaturesfoundbytherstlayer.Itprovidesamoreglobalview

28 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />

Theinputhasastronglocalstructure,soitsimpliestheproblemby<strong>using</strong> Figure18:Fullyconnected(left)andlocallyconnected(right)neuralnets.<br />

Rt<br />

Rt<br />

Rxo<br />

Rxo<br />

onlyseesapart<strong>of</strong>theinputandnotalltheinputsasinthefullyconnected 2.3.2Locallyconnectednets so-calledreceptiveelds.Itmaybeeasierfortheneuralnetifaneuron<br />

dxo<br />

dxo<br />

nets.Theneuronspecializedonitspart<strong>of</strong>theinputandcanbeusedasa localfeaturedetector.Weaddanextrahiddenlayerinordertocombine thelocalfeaturesproperly.Thepart<strong>of</strong>theinputaneuronisconnected to,iscalledareceptiveeld.Usuallyallreceptiveeldshavethesamesize Theweightkernel<strong>of</strong>therstreceptiveeldthatisconnectedtotherst andareonlyshiftedinspace(ortime)withaxedstep(xedoverlap). neuronisshowninFigure19.Wecanconstraintheweightkernelsforthe theneuralnet).Thedecrease<strong>of</strong>freedommightimprovethegeneralization variousreceptiveeldstobethesame,thisiscalledweightsharinganditis ability<strong>of</strong>thenet,butifthefreedomisreducedtoomuch,thenetoverall usedintheconvolutionalnetworks.Theadvantage<strong>of</strong>weightsharingisthe decreaseinthenumber<strong>of</strong>weightsandfreedom(andthereforecomplexity<strong>of</strong> performancemaydecrease.Themotivationforweightsharingisthatwe expectthataparticularmeaningfulfeaturecanoccuratdierenttimes(or locations)intheinput.Anexample<strong>of</strong>alocallyconnectednetisshownin<br />

Inourmodelswehavenodippinglayersandnodeviatedboreholes.In Figure18(right). 2.3.3Symmetryconstraints<br />

foracertainweightvectorw.Themirrorimage<strong>of</strong>x=(x1;x2;:::;xN)is neuralnet(inahiddennodeatthersthiddenlayer)toasignalxisf(x;w), response.Thisshouldalsoholdfortheneuralnet.Theresponse<strong>of</strong>the asignalanditsmirrorimage,asshowninFigure20,givethesametool thiscasethetoolreadingsareassumedtobesymmetric.Thismeansthat

2.3Architectureconstraints 29<br />

W<br />

11<br />

W<br />

12<br />

W<br />

31<br />

W<br />

21<br />

W<br />

22<br />

W<br />

32<br />

W<br />

31<br />

W<br />

32<br />

W<br />

33<br />

weight kernel for<br />

first receptive field<br />

n<br />

x0=(xN;:::;x2;x1).Theresponse<strong>of</strong>theneuralnettothisinputisf(x0;w). Figure19:Receptiveeldswhichoverlap13.<br />

m<br />

first receptive field<br />

first field shifted down 2/3<br />

first field shifted right 2/3<br />

Wewillnowinvestigatewhattheimplicationsareforthearchitectures<strong>of</strong> thenetswhenweforcef(x;w)f(x0;w),given Figure20:Adiscretizedinputsignalanditsmirrorimage.<br />

x1 x2 x3 . . . xn xn . . . x3 x2 x1<br />

foranyhiddennodejinthersthiddenlayer,theindexjisomitted. Forthefullyconnectednetsthefollowingequationshouldhold Here,jstandsforhiddennodej.Sincethefollowingequationsshouldhold<br />

8x2

30 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />

Thisisonlytruewhenwi=wN+1i.Thisconstraintcanbebuildinto weightscomingfrominputnodeN+1i.Theseconstraintsareshownin thenetbyforcingtheweightscomingfrominputnodeitobeequaltothe<br />

Figure21:Symmetryconstraintsonfullyconnectednet.<br />

eldsareconstrainedsymmetrically(receptiveeldfisconstrainedtoeld Ff+1forFelds). Figure21.Thesameconstraintscanbeusedforthelocallyconnectednet Figure22:Symmetryconstraintsonlocallyconnectednet.Thereceptive<br />

symmetricallyconstrained.Thismeansthatreceptiveeldfisconstrained toreceptiveeldFf+1,whereFindicatesthetotalnumber<strong>of</strong>receptive asshowninFigure22.Inthelocallyconnectednetsthereceptiveeldsare elds.Theeldsarealsointernallyconstrainedsymmetricallyasshownin Figure22. constraints.Thewaveletvector<strong>of</strong>aninputx=(x1;x2;:::;xN)(N=2j)on Forthenetsthataretrainedonwaveletcoecients,wecanalsondsome

2.3Architectureconstraints 31<br />

Figure23:Symmetryconstraintsonwaveletnets.Thecoecientsareconstrainedperdetaillevel.<br />

acertainlevelisconstructedby L(x)=k0B@x1+x2<br />

0 1<br />

x3+x4<br />

2<br />

level 3<br />

xN1+xN . 1 CA H(x)=k0B@x1x2<br />

ThecoecientsonthislevelcomefromH(x).Thecoecientsforthemirror xN1xN x3x4 . 1 CA (33)<br />

image<strong>of</strong>xarecalculatedby L(x0)=k0 B @xN+xN1 x2+x1 x4+x3 . 1CA H(x0)=k0 B @xNxN1 x2x1 x4x3 . 1CA Thecoecientsonhigherlevelsarecalculatedrecursivelyonthevector (34)<br />

signalx.Forthe2jwaveletcoecientsonlevelj>0thefollowingequation H(x0)shouldbethesameoneverydetaillevelandforeverypossibleinput shouldholdforanywaveletvectory=H(x)andy0=H(x0) <strong>of</strong>L(x).Theresponse<strong>of</strong>thenettoH(x)andtheresponse<strong>of</strong>thenetto constructedbyL(x).OneverylevelwendthatL(x0)isthemirrorimage<br />

8y2

32 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />

describesthedierencebetweentheaverageoverthersthalf<strong>of</strong>thewindow<br />

networkisshowninFigure23. positiveornegative. Onleveljwendw2j+i=w2j+1ifori=0;1;:::;2j1.Theresulting andtheaverageoverthesecondhalf<strong>of</strong>thewindow.Thetoolreadingsshould besymmetrics<strong>of</strong>orthetoolitdoesnotmatterwhetherthisdierenceis<br />

Aneuronthatislocallyconnectedextractslocalfeatures.Sometimeswe featuresthemselves(forexampleincharacterrecognition). 2.3.4Convolutionalnetwork arenotconcernedaboutthelocation<strong>of</strong>thesefeatures,butonlyaboutthe<br />

convolution subsampling convolution subsampling convolution<br />

input<br />

output<br />

neuroninthersthiddenlayertodetectaspecicfeatureatthelocation<strong>of</strong> thereceptiveeld.Wecanusethesameset<strong>of</strong>weightsforallthereceptive Theset<strong>of</strong>weights,w,belongingtoareceptiveeld,makesitpossiblefora Figure24:Convolutionalnetwork.<br />

map, all neurons<br />

this map share their<br />

wasrstonlydetectedbythersthiddenneuron,anywhereinthetotal elds.Thisenablesthersthiddenlayertodetectthatspecicfeature,that<br />

input.Theneuronsinthersthiddenlayerthatusethisset<strong>of</strong>weightsto <strong>of</strong>anumber<strong>of</strong>neuronsthatmakeuse<strong>of</strong>thesameset<strong>of</strong>weights(weight detectaspecicfeature,arecalledafeaturemap.<br />

kernel.Theconvolution,performedbyonefeaturemap,isdenedbythe sharing).Thisset<strong>of</strong>weights,asshowninFigure19,isusedasaconvolution mapstomakeitpossibletodetectmorefeatures.Eachfeaturemapconsists Thehiddenlayerisnowabletodetectonefeature.Wecanaddmorefeature<br />

sum yj=k+n1 Xkl+m1 Xlwjklxkl (36)

2.3Architectureconstraints 33<br />

wherexklistheinputpixelatlocation(k;l)andwjklistheweightbetween thisinputpixelandneuronj.Theindiceskandlindicatetheleftupper corner<strong>of</strong>thereceptiveeldandnmindicatethesize<strong>of</strong>thereceptiveeld. ceptiveeldsandsharedweights.Theoverlap<strong>of</strong>thereceptiveeldsis subsamplinglayersareliketheconvolutionlayers:theyalsomakeuse<strong>of</strong>re-<br />

Theconvolutionlayersarealternatedbyso-calledsubsamplinglayers.These<br />

isreduced.Duetothisreductioninresolution,thislayerprovidessome allatthesubsamplinglayers.(Whenthereceptiveeldsdonotoverlap maximalthisisusuallycalledsubsampling).Inthesubsamplinglayerthe spatialresolution<strong>of</strong>thefeaturemaps,generatedbytheconvolutionlayers, maximal(replacement<strong>of</strong>onepixel)attheconvolutionlayersandnotat<br />

degree<strong>of</strong>translationalandrotationalinvariance. Thistype<strong>of</strong>networkiscalledaconvolutionalnetworkandisshownin Figure24.<br />

Theprinciplesaremoreorlessthesame.Atimedelayednetworkisusedin recognition.Theone-dimensionalversioniscalledatimedelayednetwork. 2.3.5TimeDelayednetwork<br />

applicationslikespeechrecognition(Waibel,Hanazawa,Hinton,Shikano& Convolutionalnetworksareusedintwo-dimensionalproblems,likecharacter Lang1989). Theideaisthattheresponseatacertainpoint<strong>of</strong>time(ordepth)depends onpreviousinputswithacertaindelay(hencethenametimedelayed).In<br />

featuresinthenextconvolutionlayer.Thisiscalledbi-pyramidalscaling bytherstlayerispartiallycompensatedbyanincreaseinthenumber<strong>of</strong> Thisisbecausewedonotwantthetranslationalandrotationalinvariance, thatisprovidedbythesubsamplinglayers.Theloss<strong>of</strong>timeresolution thistype<strong>of</strong>nettherearenosubsamplinglayers,onlyconvolutionlayers.<br />

(Guyon1991).Thearchitectureforatypicaltimedelayednetworkisshown inFigure25.<br />

number <strong>of</strong><br />

feature<br />

maps<br />

(copies)<br />

features in time<br />

Figure25:Timedelayedneuralnetwork.<br />

feature map<br />

(shared weights within map)<br />

features<br />

(or variables)<br />

receptive field

34 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />

convolutionalnetworksandthetimedelayednetworks.Theideabehindthis 2.3.6Convolutional-regressionnetwork Thedesign<strong>of</strong>theconvolutional-regressionnetworkisbaseduponboththe<br />

copies<br />

number <strong>of</strong><br />

feature<br />

maps<br />

feature map<br />

(shared weights within map)<br />

features<br />

type<strong>of</strong>networkisthatthenetworkoutputdependsonanumber<strong>of</strong>inputs<br />

(or variables)<br />

aroundthepoint<strong>of</strong>interest.Soitdependsoninputsaboveandbelowthe loggingpoint.Forthisnetweusetheinputrepresentationproducedbythe Figure26:Convolutional-regressionnetwork.<br />

features time<br />

(uniform)discretizedslidingwindowapproach.<br />

receptive field<br />

thepreviouslayer.Allotherlayers<strong>of</strong>theconvolutional-regressionnetwork nalconvolutionalnetworks,butincontrastwiththetime-delayednetworks. Inthelatterthefeaturemapsarealwaysconnectedtoallthefeatures<strong>of</strong> some<strong>of</strong>thefeatures(variables)intheinputlayer.Thisisliketheorigi-<br />

Thefeaturemapsinthersthiddenlayercanbeconnectedtoalloronly<br />

information<strong>of</strong>theinput.ThearchitectureisshowninFigure26.Inthisexamplethefeaturemapsareconnectedtoonlyanumber<strong>of</strong>theinputfeatures.<br />

arefullyconnected.Sothisnetonlycontainsoneconvolutionlayer(and thisisdierentfromboththeconvolutionalandtime-delayedarchitectures). Weonlyuseoneconvolutionlayer,becausewewanttoretainthespatial<br />

weights)thatisneededissmall(thenumber<strong>of</strong>hiddennodesisequaltothe number<strong>of</strong>receptiveelds).Adisadvantageistheloss<strong>of</strong>spatialresolution. overconvolutionisthatthenumber<strong>of</strong>hiddennodes(andthenumber<strong>of</strong> thisisnotaconvolution,butsubsampling.Theadvantage<strong>of</strong>subsampling Thereceptiveeldsintherstlayerdonotoverlapmaximal.Soactually

2.4Errorfunction 35<br />

Afeaturemapfromthersthiddenlayerdetectsafeatureintheinput.The<br />

2.4Errorfunction hiddenlayer. secondhiddenlayerdeterminestheimportance<strong>of</strong>thelocation<strong>of</strong>thatfeature.Sothespatialinformationispreservedinthesecond(fullyconnected)<br />

Aneuralnetworkcanminimalizeanyerrorfunction.Themostcommon trainingset errorfunctionusedisthesumsquareerror,whichis,forPpatterns<strong>of</strong>the<br />

patternp. Here,dpisthedesiredoutput<strong>of</strong>patternpandapistheactualoutput<strong>of</strong> ENet=PXp=1(dpap)2 (37)<br />

Error<br />

10<br />

Relative error<br />

Network error<br />

8<br />

6<br />

4<br />

Inthisprojecttheperformance<strong>of</strong>theneuralnetismeasuredintherelative<br />

2<br />

errorbetweenthedesiredandtheactualoutput,averagedoverPpatterns Figure27:Thenetworkandrelativeerrorforxedproportionsd=a.<br />

0<br />

0 1 2 3 4 5 6 7 8 9 10<br />

ERel=vut1PPXp=1 dpap dp!2 (38)<br />

a/d<br />

Theneuralneterrorforonespecicpatternpcanbewrittenas ENet(p)=d2p 1ap dp!2 (39)

36 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />

patternswithahighdesiredoutput.Thisisnotwhatwewant,becausea smalldesiredoutputismorelikelytohaveahighrelativeerrorthanahigh errorwhenthedesiredoutputislarge.Thenetemphasizesthelearning<strong>of</strong> Thismeansthatforthesameproportiona=d,apatternpproducesahigher<br />

Beforepresentingthepatternstothenet,wehavescaledtheinputandthe targetvalue.ThuswepreferredtheerrorERelasgivenineq.38,which output.Thedesiredandactualoutputsarescaledby overcomesthisdisadvantage.<br />

followingerrorENet=PXp=1ln(dp) Theconsequence<strong>of</strong>thisscalingisthatthenetworkisminimalizingthe t0=ln(t) (40)<br />

=12PXpln2dp ln(a) 2<br />

ap (42) (41)<br />

whichisalmostthesameasminimalizingtherelativeerror,especiallywhen relativeerrorERelaredrawninFigure27. givethesameerrorsignal,asshownbytherelativeerrorinFigure27.The theproportiond=a.Forxedproportionsd=athenetworkerrorENetand Anoverestimation(a=d+k)andanunderestimation(a=dk)must theproportiond=aisalmostequalto1.Actuallythenetisminimalizing<br />

overa=d+k,becausetheformerproducesalowererrorsignal. neuralnet,withtheerrorfunctionasmentionedineq.41,favoursa=dk<br />

wewanttominimalizetherelativeerror.Whenwehadusedanotherscalingmethod(notlogarithmic),thenetworkerrorfunctionshouldhavebeen<br />

Itisveryimportanttochooseanappropriateerrorcriterion.Inthisproject targetvalues,whichisalsothecasefortherelativeerror. Itdoes,however,emphasizethelearning<strong>of</strong>smalltargetvaluesoverlarger<br />

adaptedforbetterperformance.Withthisscalinghoweverthesumsquare errorfunctionsuces.

measuredresistivityisheavilyaectedbytheinvasionresistivity(Rxo),althoughthisdependsontheinvasionradius(dxo).Beforeinvestigatingthis<br />

<strong>Forward</strong>modellinginthepresence<strong>of</strong>mudinvasionisverycomplex.The 3<strong>Forward</strong>modellingwithoutmudinvasion<br />

weonlyhaveonevariable(Rt).Theboreholeradiusanddrillinguidresistivityarexedto4.25inchand0.05m,respectively.Withonlyone<br />

complexsituation,werstlookatthecasewithoutmudinvasion.Here variabletheinputspace<strong>of</strong>earthmodelsisnottoolarge,whichallowsusto<br />

Wehavetakensmallbedsizesthatrangebetween1and5feet.Each wecanusethisinthecasewithinvasion. rstexperimentwiththeinputrepresentationandthenetworkarchitecture.<br />

earthmodelconsists<strong>of</strong>asmalllog<strong>of</strong>50feetlongand15beds,withatotal Whenwehavefoundthemostappropriaterepresentationandarchitecture,<br />

summarizedinTable2. <strong>of</strong>47models(2350feet)fortrainingand33models(1650feet)fortesting. Thetrueresistivityrangesbetween1and70m.Theseparametersare Fortheexperimentationwiththeinputrepresentationandnetworkarchitecturewehaveusedasmalltrainingset<strong>of</strong>4earthmodelsandanequallysized<br />

testset<strong>of</strong>4models.Forthesesmalltestsnoabsoluteerrorsareincluded, theaveragerelativeerror<strong>of</strong>thetrainingsetandthetestset.Anetperforms sincetheyareonlyusedforcomparison.Theperformanceismeasuredin wellwhenbotherrorsaresmallandcomparable. Table2:Parametersforearthmodelswithoutinvasion. Fixedparameters Boreholeradius Drillinguidresistivity Variables Bedsize 4.25inch<br />

Trueresistivity 1,2,3,4or5feet 1,2,3,...,70m 0.05m<br />

Data Trainingset Testset Permodel 251examples 47models 33models 50feet 15beds<br />

37

38 3.FORWARDMODELLINGWITHOUTMUDINVASION<br />

Theinputandoutputareapproximatelyscaledtotherange[1;+1](see 3.1Experimentingwithdierentscalingmethods Section2.1.3).Thisisdonebynormalization<strong>of</strong>theinputandoutput<br />

whereisthemeanandisthestandarddeviation<strong>of</strong>thevariablezonthe z0=z<br />

logarithmicscaling.Inthatcasethemeanandstandarddeviation<strong>of</strong>ln(z) trainingset.Thistype<strong>of</strong>scalingcanalsobedoneincombinationwitha (43)<br />

isused.Themeanandstandarddeviationsfortheinputandoutputare giveninthefollowingtable logarithmicscaledRt logarithmicscaledLLdorLLs3.000.83 29.0016.67 30.0016.67 3.300.80 <br />

applied,sothenormalizationonitselfisnotsucient.Whennologarithmic bestresults.Theworstresultsarefoundwhennologarithmicscalingwas thatthelogarithmicscalingincombinationwiththenormalizationgavethe andonehiddenlayer<strong>of</strong>5nodeson1004examples(4models)andfound Wetestedafullyconnectednet<strong>of</strong>25inputs(slidingwindow<strong>of</strong>5feet)<br />

errorandaminimalization<strong>of</strong>theabsoluteerrordoesnotnecessarymeana problem.Duetothisscalingtheproportionisminimalized(seeSection2.4), thedesiredandactualoutput.Wemeasuretheperformanceintherelative minimalization<strong>of</strong>therelativeerror.Thelogaritmicscalingovercomesthis scalingwasapplied,theneuralnetisminimizingtheabsoluteerrorbetween<br />

whichisalmostthesameasminimalizingtherelativeerror. Wewillusethiscombinedscalingmethodintheothertests.<br />

experimentsaredoneforawindowsize<strong>of</strong>5feet.Thetestsaredoneforboth 3.2Experimentingwiththenetworkarchitecture Nowwestartedexperimentingwiththenetworkarchitecture.Thefollowing theDeep-andtheShallow-logwith4trainingmodels(1004examples). Variation<strong>of</strong>number<strong>of</strong>hiddennodes(5,10,15,20,25).Wefound Variation<strong>of</strong>number<strong>of</strong>hiddenlayers(1or2).Twolayers<strong>of</strong>both15 thatnetswith15nodesandmoregavesimilarresults. thatincreasingthenumber<strong>of</strong>nodesupto15improvetheresults,but nodesoronehiddenlayer<strong>of</strong>15nodesgavecomparableresults.

3.3Experimentingwiththesize<strong>of</strong>theslidingwindow 39<br />

Whentheresultsarecomparablewechooseforthenetwiththefewest Addingconnections<strong>of</strong>theinputlayerdirectlytotheoutputlayer(25<br />

number<strong>of</strong>weights.Thesenetsmostlikelyhavethebestgeneralizationand thenormalfullyconnectednetwerecomparable. extraconnections).Againtheresultsfortheseextraconnectionsand<br />

thesmallesttrainingtime.Thebestarchitecturefoundbytheseexperiments<br />

Thescalingmethodandthenetworkarchitecturearexed,onlythewindow 3.3Experimentingwiththesize<strong>of</strong>theslidingwindow isanormalfullyconnectednet,withonehiddenlayer<strong>of</strong>15nodes.<br />

sizeisvariedinthefollowingtests.Wetriedwindowsizes<strong>of</strong>5,10,15and 30feet.Thesmallestwindowsizewecantryis5feet(seeSection2.1.1). Againthenetsaretrainedon4models(1004examples),onboththeDeepandtheShallow-log.Inbothcases,awindowsize<strong>of</strong>15feetgivesthebest<br />

thisdoesnotoutweighthelongertrainingtimeduetothishighnumber<strong>of</strong> weights(2281versus1156). results.Althoughtheresultsareslightlybetterforawindowsize<strong>of</strong>30feet, TheShallow-logapproximationisbetterthantheDeep-logforthesame windowsize.Wealreadyexpectedthis,becausetheDeepLaterolog\sees" alargerpart<strong>of</strong>theformationandthereforeneedsalargerwindowsize.This<br />

Thearchitecturewefoundbytheprevioustestsisafullyconnectedneural 3.4Summaryandresults wasalreadyshowninFigure3inSection1.1.1.<br />

netwithonehiddenlayer<strong>of</strong>15nodesandawindowsize<strong>of</strong>15feet(75inputs).Thisnetistrainedon47modelswithatotallength<strong>of</strong>2350feetand<br />

testedon33modelswithatotallength<strong>of</strong>1650feet.<br />

Table3:Averagerelativeerrorforearthmodelswithoutinvasion(1). Shallow-logTrainingset1.7% Deep-log Trainingset5.1% Testset 2.2% 6.0%

40 3.FORWARDMODELLINGWITHOUTMUDINVASION<br />

proximation<strong>of</strong>theShallow-log.Thegeneralizationability<strong>of</strong>thenet,that AsshowninTable3,theaveragerelativeerrorisbelow5%fortheap-<br />

istrainedontheShallow-log,isquitegood(thetesterrorandthetraining relativelylargecontributiontotheaveragerelativeerror.Therefore,wealso errorarecomparable). However,theaveragerelativeerrormaynotbeanappropriateerrorcriterion,sinceitispossiblethatafewpointswithahighrelativeerrorhavea<br />

calculatethepercentage<strong>of</strong>theoutputthathasarelativeerrorbelow5% (called\correct")asshowninTable4.Fromthistablewendthatonly Table4:Performanceonearthmodelswithoutinvasion(1). Shallow-logTrainingset98%correct Deep-log Trainingset62%correct Testset 96%correct<br />

4%<strong>of</strong>theapproximation<strong>of</strong>theShallow-loghasarelativeerrorabove5%. thelearningandgeneralizationability<strong>of</strong>thenetarequitegood. Andthisisveryclosetotheerroronthetrainingset.Therefore,wecansay 61%correct<br />

<strong>of</strong>theshoulderbeds,whichislesspr<strong>of</strong>oundinthecase<strong>of</strong>theShallow-log Theapproximation<strong>of</strong>theDeep-log,however,causesmoreproblems.This sistivityprole,incontrasttotheShallow-log.Thisiscausedbytheeect (seeSection1.1.2). isshowninFigure28.TheDeep-logdierssubstantiallyfromthetruere-<br />

TheoutputtheneuralnetgivesforthismodelisshowninFigure29.In thisgureoneseesthattheoutputgivenbythenetlooksmoreliketheresistivityprole(liketheShallow-log)andthereforehasquitealargerelative<br />

errorwithitstargetvalue(theDeep-log).Animprovementcouldbemade Weconcludethatourinputrepresentation(slidingwindowandscaling)is by<strong>using</strong>moremodelssimilartothisoneinthetrainingset.<br />

willabandonthefullyconnectednetsandlookformorecomplexarchitecturestoimproveontheresultsandtohandleinvasion.<br />

appropriatefortheproblem.Afullyconnectedneuralnetwithonehidden generalizationoverdataithasnotseenbefore.Inthefollowingchapterwe layerissucienttolearntheinput-outputmappingandtoprovidegood

3.4Summaryandresults 41<br />

Ohm meter<br />

80<br />

70<br />

True resistivity<br />

Deep-log<br />

Shallow-log<br />

60<br />

50<br />

40<br />

30<br />

Figure28:ModelthatcausesdicultiesinapproximatingtheDeep-log.<br />

20<br />

10<br />

0<br />

0 5 10 15 20 25 30 35 40 45 50<br />

depth<br />

Ohm meter<br />

80<br />

70<br />

True resistivity<br />

<strong>Neural</strong> Net<br />

Deep-log<br />

60<br />

50<br />

40<br />

30<br />

Figure29:Response<strong>of</strong>theneuralnettothedicultmodelshownabove.<br />

20<br />

10<br />

0<br />

0 5 10 15 20 25 30 35 40 45 50<br />

depth

42 4.FORWARDMODELLINGWITHMUDINVASION<br />

4<strong>Forward</strong>modellingwithmudinvasion Nowwehavefoundthatitispossibletolearnthemappingbetweenthe earthmodelsandthetoolresponse,wearegoingtolookatamoredicult (andinteresting)problem.Inthenewearthmodelsthelayers<strong>of</strong>theformationareinvadedbythedrillinguid.ThismeanswealsohavetotakeRxo,<br />

boreholeradiusandtheresistivity<strong>of</strong>thedrillinguidarexed.Theinput theinvasionresistivity,anddxo,theinvasionradius,inaccount.Againthe<br />

moreexamplestolearntheproblem. thethreevariablesRt,Rxoanddxo,ismuchlargerthanintheprevious space<strong>of</strong>earthmodels,constructedfromrandomlychosencombinations<strong>of</strong><br />

Werstperformanumber<strong>of</strong>testsonasmalltrainingsetinordertoget anideaabouthowwellthechoseninputrepresentationandarchitecture case,whentherewasonlyonevariable(Rt).Weexpectthenetworkneeds<br />

performonan(alsosmall)testset.<br />

Fixedparameters Boreholeradius Table5:Parametersforearthmodelswithinvasion.<br />

Drillinguidresistivity Variables Bedsize 1,2,3,...,20feet 0.05m 4.25inch<br />

Trueresistivity Invasionresistivity Invasionradius Data 0.5,0.7,...,2.5m 2,3,4,...,71m<br />

Trainingset Validationset A,B,C,D,E,F(M,N,O,P,Q,RandS)models 8,9,....,50inch<br />

Testset Permodel J,KandLmodels G,HandImodels 1000feet<br />

Weusebedsizesbetween1and20feet,whichismorerealisticthaninthe 5000examples 80beds<br />

training,theso-calledvalidationset.Whentheerroronthissetstartsto 19modelsfortraining,testingandvalidationasshowninTable5.During trainingonecalculatestheerroronaset<strong>of</strong>examplesthatisnotusedduring isapproximately1000feetlongandconsists<strong>of</strong>80beds.Thereisatotal<strong>of</strong> previoustests,whereweusedbedsbetween1and5feet.Eachearthmodel<br />

increase,thetrainingisstopped.Inthesmalltestswehavenotuseda

4.1Scaling<strong>of</strong>theparameters 43<br />

validationset.Thetrainingwasstopped,whenthenetworkconvergedtoa<br />

tolearnthanthemappingbetweentheearthmodelsandtheDeep-log.Ifa isminimal,buttheerroronthetestsetisnot.Intheseexperimentsweonly thatthemappingbetweentheearthmodelsandtheShallow-logwaseasier usetheShallow-log.Wehavefoundintheprevioustests,withoutinvasion, minimum.Thismeansthenetsareovertrained:theerroronthetrainingset<br />

(A)<strong>of</strong>4601samplepointsastrainingsetandonemodel(F)<strong>of</strong>4876sample Inthenextsectionssome<strong>of</strong>theresultsaregiveninduplicateinorderto getabettercomparisonbetweenthevarioustests.Wehaveusedonemodel moredicultyinlearningtheDeep-log. netndsitdiculttolearntheShallow-log,weexpectitwouldhaveeven<br />

ThetrueresistivityRtandtheShallow-andDeep-log(LLsandLLd)are 4.1Scaling<strong>of</strong>theparameters pointsastest(generalization)set.<br />

scaledasintheprevioustests.Thescalingmethodswehaveusedhereare summarizedinthefollowingtable VariableDomain Rt Rxo dxo LLd 2,3,...,71 0.5,0.7,...,2.3mnorm mlog+norm Scaling MeanDeviation<br />

LLs 8,9,...,50 inchnorm log+norm29.00 3.30 1.30 1.40 25.00<br />

0.59<br />

Inthefollowingsectionsweusetablestodescribetheperformedtests.The 1.67<br />

followingtableentriesareusedtodescribeatest:<br />

architecture:Adescription<strong>of</strong>thenetperlayerseperatedbya\.". connections:Thetype<strong>of</strong>connections.Thisisfullyorlocallyconnected(fullyconnectedbydefault).<br />

3128forexamplemeanswehave128valuespervariable.Alayer<strong>of</strong> splitintothenumber<strong>of</strong>valuespervariableorfeaturemap.Alayer<strong>of</strong> Foreachlayerthenumber<strong>of</strong>nodesisgiven.Thisnumbercanbe<br />

windowsize:Theusedsize<strong>of</strong>theslidingwindowisgiveninfeet Number<strong>of</strong>weights:Thenumber<strong>of</strong>weightsinthenetisgiven. 627meanswehave6featuremaps<strong>of</strong>27nodeseach.<br />

Samplingperiod:Theusedsamplingperiodisgiveninfeet(default valueis0.2feet). (defaultvalueis25.4feet).

44 4.FORWARDMODELLINGWITHMUDINVASION<br />

Sampling:Thetype<strong>of</strong>sampling.Thiscanbeuniform(defaultvalue), Number<strong>of</strong>epochs:Thenumber<strong>of</strong>epochsthatwasneededtotrain non-uniformornone.<br />

Generalizationerror:TheaveragerelativeerroronmodelF. Trainingerror:TheaveragerelativeerroronmodelA. thenet.<br />

4.2Experimentingwithdierentinputrepresentations<br />

inTest1.Awindowsize<strong>of</strong>25.4feetandasamplingperiod<strong>of</strong>0.2feetgives attheinputrepresentationcomingfromauniformsampledslidingwindow Because<strong>of</strong>theresultsintheexperimentswithoutinvasion,werstlooked Inthismorecomplexsituationwehavetolookatthreevariablespersamplingpoint.Thisresultsinahighnumber<strong>of</strong>inputsfortheneuralnet.<br />

us3128inputs,whichisquitehigh.ButwealsolookedataninputdescriptionbyattributesinTest2asdescribedinSection2.1.2.Theresults<br />

Sincetheslidingwindowrepresentationisgivingbetterresultsthanthe aregiveninTable6.<br />

bad.Thenetisheavilyovertrained,sincetheerroronthetestsetwasonly thenumber<strong>of</strong>weights. Thegeneralizationperformance<strong>of</strong>theattributesinputrepresentationisvery attributesdescription,wewillnowlookformethodstoreducethecomplexity<br />

41.8%after183epochs(trainingerroratthattimewas11.4%). <strong>of</strong>thenet.Thiscanbedonebyreducingthenumber<strong>of</strong>inputsorbyreducing<br />

first sampling point,<br />

inverse distance is 10<br />

0.2 feet<br />

second sampling point,<br />

inverse distance is 5<br />

Contrasts with adjacent bed<br />

Figure30:Smalldierenceininputwithattributesinputrepresentation.<br />

Input at first sampling point Rt Rxo dxo Rt/Rt2 Rxo/Rxo2 dxo/dxo2 Rt-Rt2 Rxo-Rxo2 dxo-dxo2<br />

Input at second sampling point Rt Rxo dxo Rt/Rt2 Rxo/Rxo2 dxo/dxo2 Rt-Rt2 Rxo-Rxo2 dxo-dxo2<br />

True resistivity, invasion resistivity and<br />

invasion radius<br />

Differences adjacent bed<br />

10<br />

5

4.2Experimentingwithdierentinputrepresentations 45<br />

tation.Representation Table6:Comparingdiscretizedslidingwindowagainstattributesrepresen-<br />

architecture number<strong>of</strong>weights windowsize samplingperiod Test1 3128.10.1 3861 Test2<br />

25.4feet 100.15.15.1<br />

inputdescription Uniform DiscretizedslidingAttributes 0.2feet 1771<br />

window 30feet<br />

number<strong>of</strong>epochs trainingerror generalizationerror70.0% 3060 3.1% 3484 (10beds) 4.2% >100%<br />

Table7:Comparinguniformsamplingmethodswithdierentsamplingperiods.<br />

Sampling1 architecture number<strong>of</strong>weights windowsize samplingperiod Test1 3128.10.13100.10.1 3861 25.4feet Test4<br />

number<strong>of</strong>epochs 0.2feet 3021<br />

trainingerror 3060 29.7feet<br />

generalizationerror70.0% 3.1% 0.3feet 3634 2.9% 67.3%<br />

Table8:Comparinguniformandnon-uniformsamplingmethods. Sampling2 architecture number<strong>of</strong>weights windowsize samplingperiod Test3 3128.10.10.1364.15.15.1 25.4feet 3971 29.4feet Test5<br />

Uniform 0.2feet 3151<br />

number<strong>of</strong>epochs trainingerror generalizationerror65.2% 6429 1.1% 46.4% Non-uniform 4404 1.0%

46 4.FORWARDMODELLINGWITHMUDINVASION<br />

eachbeddiersinthosecases,asshowninFigure30.Theinversedistance becomesthemostimportantvalueintheinput.Thiscanalsobeseenfrom theweights.Mostweightsarequitesmall,rangingfrom-0.5to0.5,butthe fedwithanumber<strong>of</strong>almostsimilarexamples.Onlytheinversedistancefor Apossiblecause<strong>of</strong>thebadperformanceontheattributesisthatthenetis<br />

weightsbelongingtotheconnectionsfromtheinversedistance-inputsto thehiddenlayerarelarge(1.5). isverydicult.Thenetworkisactuallypushedinsomekind<strong>of</strong>direction Itisimportanttochooseappropriateattributes,butndingtheseattributes by<strong>using</strong>attributes.Thiscanmakethelearningeasier,butitcanjustas easymakethelearningmoredicult. 4.3Experimentingwithinputreductionmethods<br />

nodesandmaybeeventwohiddenlayers<strong>of</strong>15nodes.Theformergivesus veryhigh.Forexample,awindow<strong>of</strong>29.8feetandasamplingperiod<strong>of</strong> 0.2feetproduces3150inputs.Wewouldliketouseahiddenlayer<strong>of</strong>15 Thenumber<strong>of</strong>inputscomingfromauniformsampledslidingwindowis 6781weightsandthelatter7021weights.Totrainsuchalargenetweneed approximately70000examples(Widrow'sRule<strong>of</strong>Thumbtotakeabout10 timesthenumber<strong>of</strong>freevariables).Eachexampleconsists<strong>of</strong>451values 31570000valuestotrainthisnet.Thislargesetwillslowdownthetraining (3150inputsandonetarget),sowewouldneedtostoreapproximately Itisforthisreasonthatwelookedatseveralwaystoreducethenumber<strong>of</strong> inputsforaxedwindowsize.Thedisadvantage<strong>of</strong>inputreductionisthe andcausesstorageandmemoryproblems. factthatwelooseinformation. Thenumber<strong>of</strong>inputsdependsontheusedwindowsizeandtheusedsamplingperiod.Wecoulduseasmallerslidingwindowthan25.4feet,butthis<br />

isnotanattractiveoption,becausewesuspectthetoolreadingsareaected byawindow<strong>of</strong>atleastthesize<strong>of</strong>thetool(seeSection2.1.1). 3100inputs,instead<strong>of</strong>the3150inputswitha0.2feetsamplingperiod (Test4)instead<strong>of</strong>0.2feet.Inaslidingwindow<strong>of</strong>29.7feetthisresultsin inaslidingwindow<strong>of</strong>29.8feet. Anotheroptionistouseacoarsersamplingperiod,forexample0.3feet Anotherapproachistouseanon-uniformsamplingmethod.Thisapproach<br />

4.3.1Usingdierentsamplingmethods<br />

samenetasinTest1,butwithtwohiddenlayers.Theresultsforthese only364inputsperwindow(instead<strong>of</strong>3150).InTest3weusethe uniformsamplingmethodinTest5asshowninFigure31,whichresultsin isbasedonthefactthatthephysicaltoolreceivesmost<strong>of</strong>itsinformation fromthecenterandlessfromthesides.Wehavechosenanheuristicnon-

4.3Experimentingwithinputreductionmethods 47<br />

<strong>of</strong> 29.4 feet<br />

testsareshowninTable7andTable8. Figure31:Non-uniformsamplinginslidingwindow.<br />

7.2 feet uniform sampled every 0.8 feet<br />

Whenauniformsamplingmethodisused,ititpossibletouseacoarsersamplingperiod(0.3feetinstead<strong>of</strong>0.2feet),becausethetrainingandtesting<br />

6.0 feet sampled every 0.4 feet<br />

3.0 sampled every 0.2 resultsaresimilar.Thebedboundariesaredescribedlessaccuratelywhen acoarsersamplingperiodisused.Butfromthesetestswendthatitdoes notinuencetheperformanceinanegativeway. becausethecenter<strong>of</strong>thewindowissampledwiththesamesamplingperiod asintheoriginalinput.Thenetthatistrainedonthisinputrepresentation Withthenon-uniformsamplingmethodweonlylooseaccuracyattheedges,<br />

odstoreducetheinputs. performsbetterthanthenetthatwastrainedontheoriginalinput. 4.3.2Reducingtheinputbyprojectiontoprincipalcomponents Bothmethods,coarserandnon-uniformsampling,areappropriateasmeth-<br />

WeusetheprincipalcomponentanalysisasdescribedinSection2.2.1.The eigenvectorsarecalculatedforaninputleconsisting<strong>of</strong>sixmodels.The modelsarenotalikeinternally,soitisbettertouseaset<strong>of</strong>models.Otherwisetheeigenvectorswillbeappropriateforonemodel,butnotforanother.<br />

rescaledwiththeinversenormalizationscaling<strong>of</strong>equation9,beforethe Wefoundthatitdoesnotmatterwhetheryouusesixmodelsormore,so wehavechosenforthemodelsA,M,N,P,QandR(thesemodelsarevery dissimilar).<br />

iseverywhereequaltothemean<strong>of</strong>theinputvalues.Thisrescalingisnot not100%.Thescaledvectorconsists<strong>of</strong>onlyzero's,buttherescaledvector LOIiscalculated.Duetothisscaling,theloss<strong>of</strong>informationforM=0is Bothvectors(originalN-dimensionalandprojectedM-dimensional)are<br />

absolutelynecessary,butwewantedtocalculatetheloss<strong>of</strong>informationon theoriginalinput. InTest7theoriginalinputvectorconsists<strong>of</strong>364inputs,constructedby

48 4.FORWARDMODELLINGWITHMUDINVASION<br />

Loss <strong>of</strong> information<br />

in percent<br />

50<br />

45<br />

40<br />

LOI on true resistivity Rt<br />

LOI on invasion resistivity Rxo<br />

LOI on invasion radius dxo<br />

35<br />

30<br />

25<br />

20<br />

15<br />

Figure32:Loss<strong>of</strong>informationpervariableforTest6.Originalinputconsists <strong>of</strong>3128inputs,comingfromauniformsampledslidingwindow.<br />

10<br />

5<br />

0<br />

0 8 16 24 40 48 56 64 72 80 88 96 104 112 120 128<br />

Number principal comonents used (M)<br />

principalcomponents. Table9:Comparinguniformsampledinputwithoutandwithprojectionto PCA1 architecture number<strong>of</strong>weights Test3 3128.10.10.1332.10.10.1 No 3971 Test6 1091 Yes<br />

number<strong>of</strong>epochs LOI7.4%onRt<br />

trainingerror 6429 LOI4.7%onRxo<br />

generalizationerror65.2% 1.1% 52.3% LOI4.4%ondxo 5193 1.7%

4.3Experimentingwithinputreductionmethods 49<br />

Loss <strong>of</strong> information<br />

in percent<br />

50<br />

45<br />

40<br />

LOI on true resistivity Rt<br />

LOI on invasion resistivity Rxo<br />

LOI on invasion radius dxo<br />

35<br />

30<br />

25<br />

20<br />

15<br />

Figure33:Loss<strong>of</strong>informationpervariableforTest7.Originalinputconsists <strong>of</strong>364inputs,comingfromanon-uniformsampledslidingwindow.<br />

10<br />

5<br />

0<br />

0 8 16 24 32 40 48 56 64<br />

Number principal comonents used (M)<br />

tiontoprincipalcomponents. Table10:Comparingnon-uniformsampledinputwithoutandwithprojec-<br />

PCA2 architecture number<strong>of</strong>weights windowsize samplingperiod Test5 364.15.15.1332.15.15.1 29.4feet 3151 Test7<br />

PCA Non-UniformNon-Uniform<br />

1711 29.4feet Yes<br />

number<strong>of</strong>epochs LOI7.2%onRt<br />

trainingerror 4404 LOI4.2%onRxo<br />

generalizationerror46.4% 1.0% LOI3.9%ondxo 3896 1.4% 45.6%

50 4.FORWARDMODELLINGWITHMUDINVASION<br />

TheLOI,forbothtests,iscalculatedforP=1520patternscomingfrom modelsJ,OandS. values<strong>of</strong>MisshowninFigure33. anon-uniformsampledslidingwindowasinTest5.TheLOIforvarious<br />

forthetrainingandtestset. Inthefollowingtables,Table9andTable10,theLOIisgivenpervariable<br />

weightsarereducedwith73%and46%respectively. weachieveaninputreduction<strong>of</strong>75%andinthesecondtest50%.The Inbothtestswendthatthetrainingandgeneralizationerrorforthetests analysisisagoodmethodtoreducethenumber<strong>of</strong>inputs.Inthersttest withandwithoutPCAarecomparable.Thismeanstheprincipalcomponent<br />

4.3.3Reducingtheinputsbyremovingwaveletcoecients originalinputs.Wedonotknowhowmuchcoecientscanberemovedon Inthefollowingtestsweusewaveletcoecientsasinputsinstead<strong>of</strong>the thenerdetaillevelssothatnottoomuchinformationislost.Therefore,<br />

Therstinterestingaspectisthatthenetlearnsbetterwhencoecientsare TheresultsaregiveninTable11andTable12. thecoecientsonthenestdetaillevelandsomeonhigherdetaillevels. coecientsareremovedatallandinthelasttest,Test11,weremoveall werunseveraltestswithdierentreductions.Inthersttest,Test8,no<br />

<strong>of</strong>thebadperformance<strong>of</strong>thenetinTest8. inputnodejis0,thereisnoweightcorrection.Thisisprobablythecause localgradientandmostimportantontheinputsignal<strong>of</strong>nodej.Sowhen latedasindicatedbyequation7.Itdependsonthelearningparameter,the removed.Duringtrainingtheweightcorrectionforahiddennodeiscalcu-<br />

becausethedierencebetweenthersthalfandthesecondhalf<strong>of</strong>(apart Thewaveletcoecientsare0whenthereisnobedtransitionatthatspot, becausethereareonlyafewbedboundariesinthewindow.Sometimesa <strong>of</strong>)thewindowis0.Most<strong>of</strong>thecoecientsonhigherdetaillevelsarezero, bedboundaryisnotevendetectedasshowninFigure34. Onthenestdetaillevelthisproblemisquitesevere,becausethesecoecientsonlyseeaverysmallpart<strong>of</strong>theinputandthissmallpartisnotlikely<br />

hiddennodedoesnot\learn"fromthiscoecient. fromthiscoecienttothehiddennodeisnotupdated.Inotherwords,the tocontainabedboundary.Wheneverthebedboundaryisnotdetectedor whenthereisnobedboundaryatall,thecoecientsarezeroandtheweight

4.3Experimentingwithinputreductionmethods 51<br />

Table11:Comparinginputrepresentationswithdierentnumber<strong>of</strong>wavelet coecients(1). Haar1 architecture number<strong>of</strong>weights coecientreductionnone Test8 3128.10.1368.15.1 3861 Test9<br />

number<strong>of</strong>epochs 3091<br />

trainingerror 2125 220onlevel6<br />

generalizationerror63.5% 4.1% 210onlevel5 4303 0.8% 51.5%<br />

Table12:Comparinginputrepresentationswithdierentnumber<strong>of</strong>wavelet coecients(2). Haar2 architecture number<strong>of</strong>weights coecientreduction225onlevel6232onlevel6 Test10 348.10.1 2191 212onlevel5212onlevel5 Test11<br />

23onlevel423onlevel4 334.15.1 1561<br />

number<strong>of</strong>epochs trainingerror generalizationerror50.3% 4479 1.0% 56.4% 3848 0.9%

52 4.FORWARDMODELLINGWITHMUDINVASION<br />

b<br />

Part <strong>of</strong> input signal that lies<br />

in sliding window, wavelet<br />

a<br />

coefficients on this level<br />

are (0,(b-a)/2,0,0)<br />

b<br />

a<br />

Input signal after sliding window<br />

is moved 0.2 feet to the right;<br />

wavelet coefficients on this level<br />

are (0,0,0,0)<br />

discretized input values<br />

1 2 3 4 5 6 7 8<br />

Thereareapproximately2bedboundariesperpattern.Notallboundaries Figure34:Notalltransitionsaredetectedbywaveletcoecients.<br />

aredetectedbythecoecientsonthenestdetaillevel,butwhenitisnot detectedbycoecientciitwillbedetectedbycoecientci1aftermoving<br />

coefficients<br />

theslidingwindow.Soonaverage50%<strong>of</strong>theboundariesisdetectedon thenestdetaillevel.Thetrainingsetconsists<strong>of</strong>4601patterns,sothis meanstheweightcomingfromcoecientcitohiddennodehjisupdated approximately72times(4601patterns,2boundariesperpatternfromwhich 50%isdetectedand64coecients).Ausualtrainingtakesabout2000to<br />

WhenwecomparetheresultstothenettrainedinTest1,weseethatthe thesenetsismuchbetterthantheperformance<strong>of</strong>thenetinTest8. detaillevelandals<strong>of</strong>romhigherlevelsandweseethattheperformance<strong>of</strong> InTest9,10and11weremoveanumber<strong>of</strong>thecoecientsonthenest 3000updatesperweight.<br />

lessweights.Thenumber<strong>of</strong>trainingexamplesinproportiontothenumber forthebettergeneralizationperformance,isthatthenetscontainmuch <strong>of</strong>73%ispossible,withoutreducingthenetperformance.Oneexplanation inputsisreducedfrom3128to368,348and334.Aninputreduction netshavebettertrainingandgeneralizationerrors,althoughthenumber<strong>of</strong><br />

analysis<strong>of</strong>Test6and7. improvementingeneralizationisbetterthanfortheprincipalcomponent andgeneralizationmorethantheoriginaldiscretizedslidingwindow.The possiblethatthenetndsthisinputrepresentationfacilitatesthetraining <strong>of</strong>weightsishigherintheseteststhaninTest1.Butitcouldalsobe

4.4Creatingamorerepresentativetrainingset 53<br />

4.4Creatingamorerepresentativetrainingset Ourtrainingsetisnotveryrepresentative,becausetheexamples<strong>of</strong>adjacent loggingpointslookverysimilar.Thisiscausedbythefactthatweusea<br />

logandsecondlyby<strong>using</strong>\dicult"(non-similar)examples. ited).Therearetwomethodsweemploytoincreasetherepresentativeness <strong>of</strong>thetrainingset:rstlyby<strong>using</strong>acoarsersamplingperiodforthetarget becausewewantasmuchexamplesperlogaspossible(thedatasetislim-<br />

densesamplingperiodforthetargetlog.Weusethisdensesamplingperiod<br />

trainingset<strong>of</strong>thesamesizeasouroriginaltrainingsetweuse15models ducesabout300examples.ThisiswhatwedidinTest12.Tocreatea withthissamplingperiod.Thistrainingsetismorerepresentative,because Whenweuseasamplingperiod<strong>of</strong>3feet,onemodel<strong>of</strong>1000feetonlypro-<br />

thenetwork. itcontainsmoredierentexamples.Weexpectthiswillmakeitmoredif-<br />

Aftertrainingthenet,wehavelookedattheapproximation<strong>of</strong>thenettoan cultforthenettolearntheproblem.Thenetworkhastondamore<br />

arbitrarylog.Wefoundthatthehighestrelativeerrorsoccuraroundareas generalsolutiontotalltheexamples.Thisimprovesthegeneralization<strong>of</strong><br />

wouldbemorerepresentativewhenitcontainedmoreexampleslikethis. Actuallythisissomekind<strong>of</strong>weighing<strong>of</strong>theexamples.Byoeringmore <strong>of</strong>lowresistivityandareaswithhighresistivitycontrasts.Thetrainingset<br />

theotherparts<strong>of</strong>logA(withatotal<strong>of</strong>6036examplesastrainingset).The \dicult"examples,theseexampleswillgetmoreattentionintheminimalizationprocess.InTest13weuseatrainingset<strong>of</strong>4600examples,coming<br />

fromdicultparts<strong>of</strong>logA,PandSand1436coarsersampledexamples<strong>of</strong> resultsforthesetestsaregiveninTable13andTable14. Aswecanseefromthesetests,bothmethodsimprovethegeneralization performance<strong>of</strong>thenet.Thetraining,however,becomesmorecomplicated. dierentexamplesthatlooklikeexamplesthatarefoundinreality. Itisveryimportanttocreateatrainingsetthatcontainsalargenumber<strong>of</strong>

54 4.FORWARDMODELLINGWITHMUDINVASION<br />

Table13:Comparingtrainingset<strong>of</strong>targetlogsandtrainingset<strong>of</strong>coarser sampledtargetlogs. Trainingset1 architecture number<strong>of</strong>weights trainingset number<strong>of</strong>epochs Test1 3128.10.13128.10.1 3861 modelA Test12<br />

trainingerror generalizationerror70.0% 3060 3.1% 3861 mixture<strong>of</strong>15models 3023 5.6% 26.9%<br />

parts<strong>of</strong>targetlogs. Table14:Comparingtrainingset<strong>of</strong>targetlogsandtrainingset<strong>of</strong>dicult Trainingset2 architecture number<strong>of</strong>weights trainingset Test1 3128.10.13128.10.1 3861 Test13<br />

number<strong>of</strong>epochs modelA 3861<br />

trainingerror 3060 dicultexamples<strong>of</strong><br />

generalizationerror70.0% 3.1% modelsA,PandS 1042 9.8% 30.3%

4.5Intermediateresultsinputrepresentations 55<br />

4.5Intermediateresultsinputrepresentations<br />

validationset(14153examples)andthemodelsJ,K,L,M,N,O,P,Q,R Weselectsome<strong>of</strong>thenetsfromthepreviousteststotrainthemonalarge<br />

andSastestset(47485examples). thetrainingsetcontains28331examples.WeusethemodelsG,HandIas modelsA,B,C,D,EandF.Thetargetlogsaresampledevery0.2feet,so trainingset.ThetrainingsetfortheShallow-logisconstructedfromthe<br />

themodelsthenetistrainedon,theothermodelsareusedforvalidation andtesting.Therstnet,Non-uniform,isanetwithanon-uniformsampledinputandPCAreductionfrom364inputsto35inputsforRtand<br />

InFigure35theresultsforthreenetsareshown.Therstsixmodelsare<br />

net,UniformfromTest4,isauniformsampledinputwithawindow<strong>of</strong> some<strong>of</strong>thecoecientsfrom3128inputsto368inputs.Andthethird 29.7feetandasamplingperiod<strong>of</strong>0.3feet,resultingin3100inputs. netwithHaarcoecientsasinputs.Theinputsarereducedbyremoving 20inputsforbothRxoanddxo.Thesecondnet,HaarfromTest9,isa<br />

InFigure35thepercentage<strong>of</strong>thelogthathasarelativeerrorbelow5%is andtestingset. resultsonthetrainingset,buttheyallperformequallyoverthevalidation shown.<br />

Themethodstoreducethenumber<strong>of</strong>inputsworkverywell.Thenetworks AscanbeseeninFigure35,thenon-uniformsampledinputgivesthebest<br />

areabletolearntheproblemevenwhenalargenumber<strong>of</strong>theinputsis removed.Thegeneralizationperformance,however,isnotsucientandwe havealsolostinformation.Inthenextsectionswewillinvestigatecertain architecturalconstraintsinordertoimproveonthegeneralizationperformance.

56 4.FORWARDMODELLINGWITHMUDINVASION<br />

percentage <strong>of</strong> log<br />

that is correct<br />

100<br />

95<br />

90<br />

85<br />

80<br />

75<br />

70<br />

Non-Uniform<br />

Haar<br />

Uniform<br />

65<br />

Figure35:Intermediateresults<strong>of</strong>trainingthreedierentneuralnetson6<br />

60<br />

models.<br />

55<br />

50<br />

45<br />

training validation testing

4.6Experimentingwitharchitectureconstraints 57<br />

4.6Experimentingwitharchitectureconstraints Intheprevioustestsweonlyusedfullyconnectednets.Nowwewilllook constraint,becauseitcouldalsomakethelearningmoredicultanddecreasethegeneralizationperformance.Thisoccurswhentheweightsharing<br />

onlyhelpthetrainingandgeneralizationwhentheweightsharingisalogical consequence<strong>of</strong>theproblem.<br />

atlocallyconnectednetsandotherconnectionconstraints.Anattractive methodtoreducethenumber<strong>of</strong>weightsisbyforcingcertainweightsto isforcingthenettondasolutionthatdoesnotttheproblem.Itwill beequal.Ofcoursewehavetobeverycarefulwhenweusethistype<strong>of</strong><br />

sucientnumber<strong>of</strong>(dierent)examplesisgiven.Wetraindierentfully 4.6.1Experimentingwithfullyconnectednets Aswehaveseen,thefullyconnectednetscanbetrainedquitewellwhena<br />

feasibletotrainanetwiththatmanyweightsonalargetrainingset. Wendthattwohiddenlayersimprovethetrainingresults,butitisnot inTest1andTest3.Theresults<strong>of</strong>thesetestsaregiveninTable15. connectednetswithuniformsampledinputsandoneortwohiddenlayers<br />

4.6.2Experimentingwithlocallyconnectednets Theadvantage<strong>of</strong>locallyconnectednetsoverfullyconnectednetsisthatthey needmuchlessweightsandtheneuronsinthehiddenlayercanspecializeon theirpart<strong>of</strong>theinput.Thesizeandoverlap<strong>of</strong>thereceptiveeldsarethings<br />

netfromTest16tothefullyconnectednetfromTest3,wehaveobtained tobedetermined.WehavetriedtwolocallyconnectednetsinTest15and<br />

aweightreduction<strong>of</strong>63%.Althoughthetrainingerrorisslightlyworse, Test16.TheresultsaregiveninTable16. number<strong>of</strong>hiddennodesonthesecondhiddenlayer.Whenwecomparethe Theresultsarebetterwhenweusesmallreceptiveeldsandasucient<br />

locallyconnectednetsaremoreattractive(inthisproblem)thanthefully thegeneralization,whichismoreimportant,isalotbetter.Wendthat connectednets.<br />

inSection2.3.3,inthefullyconnectednet(Test17andTable17),thelocallyconnectednet(Test18andTable18)andthewaveletnets(Test19<br />

Inthefollowingtestswehavebuildthesymmetryconstraints,asdescribed 4.6.3Usingsymmetryconstraints<br />

andTable19).

58 4.FORWARDMODELLINGWITHMUDINVASION<br />

Fullyconnected architecture number<strong>of</strong>weights number<strong>of</strong>epochs Table15:Comparingdierentfullyconnectednets.<br />

trainingerror Test1 3128.10.13128.10.10.1 3861 3060 Test3<br />

generalizationerror70.0% 3.1% 3971 6429 1.1% 65.2%<br />

Locallyconnected connections architecture number<strong>of</strong>weights Table16:Comparingdierentlocallyconnectednets.<br />

receptiveeldsize Test15 LocallyconnectedLocallyconnected 3128.12.5.1 1523 3128.19.15.1 Test16<br />

receptiveeldoverlap80% number<strong>of</strong>epochs 7.8feet 1475<br />

trainingerror 2679 3.8feet<br />

generalizationerror 7.5% 61.7% 48.9% 70% 5267 2.1%<br />

Table17:Comparingfullyconnectednetswithoutandwithsymmetryconstraints.Symmetry1<br />

architecture number<strong>of</strong>weights symmetryconstraintsNo 3128.10.13128.10.1 3861 Test1 1941 Yes Test17<br />

number<strong>of</strong>epochs trainingerror generalizationerror 3060 70.0% 3.1% 655 6.6% 37.7%

output node<br />

second hidden layer<br />

<strong>of</strong> 15 nodes<br />

total<br />

number <strong>of</strong><br />

6 feature<br />

maps<br />

(copies)<br />

feature map<br />

(shared weights within map)<br />

4.6Experimentingwitharchitectureconstraints 59<br />

performance.Thisidea<strong>of</strong>symmetryconstraintsisworkedoutfurtherin thetrainingmoredicultforthenet,butitimprovesthegeneralization theconvolutional-regressionnets. Inallteststhenumber<strong>of</strong>weightsisreducedapproximately50%.Thismakes<br />

4.6.4Experimentingwithconvolutionalregressionnets<br />

fromthersthiddenlayeranddeterminesthelocation<strong>of</strong>thefeatureinthe Intheconvolutional-regressionnetsweconstrainallreceptiveeldstoshare input. racyinspatialresolution.Thesecondhiddenlayercombinestheactivations thesameweights.Thesizeandtheoverlap<strong>of</strong>theeldsdeterminetheaccu-<br />

Rt<br />

denlayerareconnectedtoallthevariablesintheinputlayer. Figure36:Convolutional-regressionnet.Thefeaturemapsinthersthid-<br />

Rxo<br />

dxo<br />

Thenumber<strong>of</strong>featuremapsdeterminehowmanyfeaturescanbedetected.<br />

window size<br />

dependuponthenumber<strong>of</strong>featuremapsused(for15hiddennodesin Weshouldnotuseto<strong>of</strong>ewfeaturemaps,otherwisethenethasdicultyin learningtheproblemandingeneralization.Thenumber<strong>of</strong>weights,however,<br />

29.8 feet<br />

receptive field

60 4.FORWARDMODELLINGWITHMUDINVASION<br />

constraints. Table18:Comparinglocallyconnectednetswithoutandwithsymmetry Symmetry2 connections architecture number<strong>of</strong>weights receptiveeldsize Test16 LocallyconnectedLocallyconnected 3128.19.15.1 1475 3128.19.15.1 Test18<br />

receptiveeldoverlap70% symmetryconstraintsNo 3.8feet 905<br />

number<strong>of</strong>epochs Yes 3.8feet<br />

trainingerror 5267 70%<br />

generalizationerror 2.1% 48.9% 37.8% 5536 2.3%<br />

Table19:Comparing\wavelet"netwithoutandwithsymmetryconstraints. Symmetry3 architecture number<strong>of</strong>weights remarks Test9 368.15.1 3091 Test19<br />

symmetryconstraintsNo WaveletcoecientsWaveletcoecients Reduction20,10 Reduction20,10 Yes 367.15.1 1561<br />

number<strong>of</strong>epochs trainingerror generalizationerror 4303 0.8% 51.5% 46.6% 3137 3.6%

4.6Experimentingwitharchitectureconstraints 61<br />

second hidden layer<br />

<strong>of</strong> 15 nodes<br />

total<br />

number <strong>of</strong><br />

9 feature<br />

maps<br />

copies<br />

(3 feature maps)<br />

feature map<br />

(shared weights within map)<br />

Figure37:Convolutional-regressionnet.Thersthiddenlayerconsists<strong>of</strong><br />

Rt<br />

Rxo<br />

dxo<br />

weightsbyk15).Atrade-oshouldbemadebetweenthesizeandoverlap thesecondhiddenlayeronefeaturemap<strong>of</strong>knodesincreasesthenumber<strong>of</strong> toone<strong>of</strong>thevariablesintheinputlayer. threesets<strong>of</strong>featuremaps.Eachsetconsists<strong>of</strong>threemapsandisconnected<br />

window size<br />

<strong>of</strong>theelds,thenumber<strong>of</strong>featuremapstouseandtheresultingnumber<br />

29.8 feet<br />

receptive field<br />

ontheotherhand,thersthiddenlayerwassplitintothreevariablemaps, toallthevariables<strong>of</strong>theinputlayerasshowninFigure36.InTest23, itscorrespondingvariableintheinputlayer.ThisisshowninFigure37. eachcontainingthreefeaturemaps.Eachvariablemapwasconnectedto <strong>of</strong>weightsinthenet.InTest20,21and22thehiddenlayerwasconnected<br />

TheresultsforthesetestsaregiveninTable20andTable21.Forallthese testset.Thegeneralizationperformance<strong>of</strong>thesenetsissignicantlybetter tetsthenetsarelocallyconnectedandthewindowsizeis29.8feet.The convolutional-regressionnetsperformverywellonboththetrainingandthe<br />

generalizationisapproximately2.5timesbetter. thanforallthenetswehavetriedbefore.Whenwecomparetheresults nectednetwithtwolayers),wehaveaweightreduction<strong>of</strong>29%andthe fromTest21withtheresultsfromTest3forexample(Test3isafullycon-

62 4.FORWARDMODELLINGWITHMUDINVASION<br />

thebedboundaries.Itsmoothstheinputsignalandtheactivationsare InFigure39theactivations<strong>of</strong>thersthiddenlayeratacertaindepthare Fromtheseactivationswecanseethersthiddenlayerisactuallydetecting plotted.TheseactivationscomefromTest21. Wehaveinvestigatedthefeaturesthataredetectedbythersthiddenlayer.<br />

constantforparts<strong>of</strong>theinputsignalthatareconstant.

4.6Experimentingwitharchitectureconstraints 63<br />

Convolutional1 architecture number<strong>of</strong>weights samplingperiod Table20:Resultsforconvolutional-regressionnets(1).<br />

receptiveeldsize Test20 360:69:5:13150:627:15:1 503 0.5feet Test21<br />

receptiveeldoverlap75% 5.5feet 2827<br />

gure 0.2feet<br />

number<strong>of</strong>epochs Figure36 3.8feet<br />

trainingerror 3316 75%<br />

generalizationerror 3.9% 34.7% Figure36 2865 0.6% 25.5%<br />

Convolutional2 architecture number<strong>of</strong>weights samplingperiod Table21:Resultsforconvolutional-regressionnets(2).<br />

receptiveeldsize Test22 3100:616:15:13150:927:15:1 1927 0.3feet Test23<br />

receptiveeldoverlap80% 7.2feet 0.2feet 3.8feet 3865<br />

gure number<strong>of</strong>epochs trainingerror Figure36 4081 Figure37 75%<br />

generalizationerror 1.6% 26.3% 0.5% 28.6% 3689

64 4.FORWARDMODELLINGWITHMUDINVASION<br />

Ohm meter<br />

70<br />

Rt<br />

60<br />

50<br />

40<br />

30<br />

Figure38:Trueresistivityprole<strong>of</strong>modelAatdepth615feet.<br />

20<br />

10<br />

0<br />

600 605 610 620 625 630<br />

depth<br />

activation<br />

1<br />

0.8<br />

0.6<br />

Feature map 1<br />

Feature map 2<br />

Feature map 3<br />

Feature map 4<br />

Feature map 5<br />

Feature map 6<br />

0.4<br />

0.2<br />

0<br />

-0.2<br />

-0.4<br />

-0.6<br />

6featuremaps<strong>of</strong>27nodeseach. TheseactivationsareformodelAatdepth615feet.Thislayerconsists<strong>of</strong> Figure39:Activation<strong>of</strong>rsthiddenlayerperfeaturemapforTest21.<br />

-0.8<br />

-1<br />

5 10 20 25 30 35<br />

node

4.7Intermediateresultsarchitecturedesign 65<br />

Wetrainaconvolutional-regressionnetonalargerset<strong>of</strong>models.The 4.7Intermediateresultsarchitecturedesign previousnetworks(28331,14153and47485patternsrespectively).Thenet trainingset,validationsetandtestsetisequaltotheoneusedforthe thatisusedisshowninFigure36.Theperformanceincomparisonwiththe netismuchbetterthanforthepreviousnets. Asonecansee,thegeneralizationperformance<strong>of</strong>theconvolutional-regression previousresultsisshowninFigure40. Wearegoingtousethistype<strong>of</strong>netfortheapproximation<strong>of</strong>boththeDeepandtheShallow-log.<br />

theconvolutional-regressionnets.WetrainanetontheShallow-logand oneontheDeep-log.Weusetheconvolutional-regressionnetfromTest21. Thenetswiththebestperformance(trainingandgeneralizationerror)are 4.8Summaryandresults<br />

Thisnethasaslidingwindow<strong>of</strong>29.8feet,sampledevery0.2feet(3150 inputs),27receptiveelds<strong>of</strong>3.8feetwith75%overlapand6featuremaps. Weperformthefollowingexperiments: Shallow-log(1):Wetrainaconvolutional-regressionnetonearth<br />

Table22:Averagerelativeerrorandperformanceforearthmodelswith aresampledevery0.2feet.TheresultsaregiveninTable22. modelswithinvasion(modelsA,B,C,D,EandF).Thetargetlogs<br />

invasion(Shallow-log). Trainingset Validationsetrelativeerrorpredictedcorrect averagepercentage<strong>of</strong>setnumber<strong>of</strong><br />

Testset 4.3% 5.1% 6.2% 82%correct 77%correct 88%correct patterns 14153 47485 28331<br />

Shallow-log(2):Afterthistrainingweaddmodelswithoutinvasion andsomemodelswithinvasionandrestartthetraining.Forthis arenowsampledevery1.0feet)and40modelswithoutinvasion(target purposeweusethewholetrainingset(seeTable5)(thetargetlogs logswersampledevery0.4feet).TheresultsareshowninTable23. Allmodelscontaineitherinvasionornot.Therefore,theperformance issplitintoinvasionandnoinvasion.

66 4.FORWARDMODELLINGWITHMUDINVASION<br />

percentage <strong>of</strong> log<br />

that is correct<br />

100<br />

95<br />

90<br />

85<br />

80<br />

75<br />

70<br />

Non-Uniform<br />

Haar<br />

Uniform<br />

Convolutional-regression<br />

65<br />

Figure40:Intermediateresults(2)<strong>of</strong>trainingdierentneuralnetson6<br />

60<br />

models.<br />

55<br />

50<br />

Table23:Averagerelativeerrorandperformanceforearthmodelswithand<br />

45<br />

withoutinvasion(Shallow-log).<br />

training validation testing<br />

relativeerrorpredictedcorrect averagepercentage<strong>of</strong>setnumber<strong>of</strong> 5.3% 76%correct patterns<br />

noinvasionTrainingset Validationset 5.5% 75%correct 61688<br />

Testset 5.6% 77%correct 74%correct 14153 14128<br />

10040

4.8Summaryandresults 67<br />

Deep-log:Wetrainaconvolutional-regressionnetonbothmodels<br />

Table24:Averagerelativeerrorandperformanceforearthmodelswithand withandwithoutinvasion.Weusethesametrainingsetasforthe<br />

withoutinvasion(Deep-log). Shallow-log.TheresultsaregiveninTable24.<br />

relativeerrorpredictedcorrect averagepercentage<strong>of</strong>setnumber<strong>of</strong> 8.4% 52%correct patterns<br />

noinvasionTrainingset Validationset 8.8% 9.0% 51%correct 50%correct 61688<br />

Testset 7.0% 6.9% 56%correct 58%correct 14153 14128<br />

scratch.ThenetthatwastrainedontheShallow-logwasoptimalformodels Thelasttwonetsarenotoptimalinasensethattheyarenottrainedfrom10040<br />

invasion.ThenetthatwastrainedontheDeep-logusedtheoptimalset<strong>of</strong> withinvasion.Thenweaddedmodelswithoutinvasionandnewmodelswith weightsfromtherstnetthatwastrainedontheShallow-log(thiswas weights)anddirectly<strong>using</strong>alargetrainingset. resultscanbeachievedbyretrainingthenets(randominitialization<strong>of</strong>the andcontinueditstrainingonnewmodelswithandwithoutinvasion.Better donebecausetheShallow-logandDeep-loglookverysimilarinpractice)

68 5.THENEURALNETWORKASFASTFORWARDMODEL<br />

5Theneuralnetworkasfastforwardmodel wardmodels.Theaccuracyisnotyetsucient,butcanbeimprovedby Inthissectionwepresenttheneuralnetworksthatcanbeusedasfastfor-<br />

trainingonamorerepresentativetrainingset.Foreachnetthe\inputdomain"isgiven.Thisisadescription<strong>of</strong>themodelsthenetistrainedon.<br />

Forearthmodelssimilartothesetrainingmodels,thenetgivesanapproximationwithanaccuracythatisalsogiveninthisdescription.Whenan<br />

<strong>of</strong>(apart<strong>of</strong>)theearthmodel(onlyformodelswithinvasion),theapproximation<strong>of</strong>thelogbytheneuralnetandacumulativeerrorplot.Inthis<br />

cumulativeerrorplotthepercentage<strong>of</strong>thelogthathasarelativeerrorbelowR%(forR=0:::25)isgiven.Inallplotstheinvasionradiusisgivenin<br />

earthmodelliesoutsidethisdomain,onecannotexpectthenettoperform optimal. Foreachnetthebestandworsttestresultisgiven.Theseresultsconsist<br />

Adescription<strong>of</strong>thisnetcanbefoundinChapter3.Thenetisfullyconnected.TheinputdomainforthenetthatistrainedontheDeep-andthe<br />

net toolresponse sizeslidingwindow sizetrainingset rangeRt rangebedsize Deep-logandShallow-log<br />

5.1Applicationtoearthmodelswithoutinvasion 0.1inchandtheresistivitiesinm.<br />

Shallow-log,herecalledNoInvasion,isgiveninthefollowingtable:<br />

speed 1...70m 1...5feet 11797examples 400pnt/sec 15feet<br />

InFigures41and42theworstandbesttestresultsareshownforthe accuracyDeep-log accuracyShallow-log 2.2%<br />

Deep-logapproximation<strong>of</strong>thenetNoInvasion.Fromthecumulativeerror 6.0%<br />

plotsfortheseapproximationsonecanseethatthedeviationinperformance 5%,butinthebestcasethisis100%. isquitelarge.Intheworstcaseonly8%<strong>of</strong>theloghasarelativeerrorbelow InFigures43and44theworstandbesttestresultsareshownforthe manceismuchlessthaninthecase<strong>of</strong>theDeep-log.Intheworstcase75% Shallow-logapproximation<strong>of</strong>thenetNoInvasion.Thedeviationinperfor-<br />

<strong>of</strong>theloghasarelativeerrorbelow5%andinthebestcasethisis100%.

5.1Applicationtoearthmodelswithoutinvasion 69<br />

100<br />

True resistivity Rt<br />

Deep-log<br />

Approximation by neural net<br />

10<br />

1<br />

0 5 10 15 20 25 30 35 40 45 50<br />

Percentage <strong>of</strong> log<br />

with relative error<br />

below R %<br />

100<br />

90<br />

80<br />

70<br />

60<br />

50<br />

40<br />

30<br />

Figure41:Worstcaseneuralnetapproximation<strong>of</strong>Deep-log.Averagerelativeerroris14.2%.<br />

20<br />

10<br />

0<br />

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />

Relative error R

70 5.THENEURALNETWORKASFASTFORWARDMODEL<br />

100<br />

True resistivity Rt<br />

Deep-log<br />

Approximation by neural net<br />

10<br />

1<br />

0 5 10 15 20 25 30 35 40 45 50<br />

Percentage <strong>of</strong> log<br />

with relative error<br />

below R %<br />

100<br />

90<br />

80<br />

70<br />

60<br />

50<br />

40<br />

30<br />

Figure42:Bestcaseneuralnetapproximation<strong>of</strong>Deep-log.Averagerelative erroris2.6%.<br />

20<br />

10<br />

0<br />

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />

Relative error R

5.1Applicationtoearthmodelswithoutinvasion 71<br />

100<br />

True resistivity Rt<br />

Shallow-log<br />

Approximation by neural net<br />

10<br />

1<br />

0 5 10 15 20 25 30 35 40 45 50<br />

Percentage <strong>of</strong> log<br />

with relative error<br />

below R %<br />

100<br />

90<br />

80<br />

70<br />

60<br />

50<br />

40<br />

30<br />

Figure43:Worstcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris8.5%.<br />

20<br />

10<br />

0<br />

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />

Relative error R

72 5.THENEURALNETWORKASFASTFORWARDMODEL<br />

100<br />

True resistivity Rt<br />

Shallow-log<br />

Approximation by neural net<br />

10<br />

1<br />

0 5 10 15 20 25 30 35 40 45 50<br />

Percentage <strong>of</strong> log<br />

with relative error<br />

below R %<br />

100<br />

90<br />

80<br />

70<br />

60<br />

50<br />

40<br />

30<br />

Figure44:Bestcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris1.2%.<br />

20<br />

10<br />

0<br />

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />

Relative error R

5.2Applicationtoearthmodelswithinvasion 73<br />

Shallow-log.Itistrainedonearthmodelsasdescribedinthefollowingtable: regressionnet.Thefollowingnetonlyproducesanapproximation<strong>of</strong>the Adescription<strong>of</strong>thisnetcanbefoundinChapter4.Thisisaconvolutional-<br />

5.2Applicationtoearthmodelswithinvasion<br />

net toolresponse sizeslidingwindow sizetrainingset rangeRt Invasion<br />

rangeRxo rangedxo 0.5...2.5m 1...71m 8...50inch 28331examples 30feet<br />

rangebedsize speed 1...20feet<br />

InFigure45two<strong>of</strong>thetestmodelsareshown.Themodelatthetopis accuracyShallow-log 6.2% 90pnt/sec<br />

Inthecumulativeerrorplot<strong>of</strong>therstapproximationisshownthat70%<strong>of</strong> thelogisapproximatedwitharelativeerrorbelow5%.Inthecumulative proximation<strong>of</strong>thenetInvasionareshown. dicultfortheneuralnetandthemodelatthebottomrelativelyeasy.In Figures46and47theworstandbesttestresultsfortheShallow-logap-<br />

errorplot<strong>of</strong>thesecondapproximationonecanseethatinthebestcase 83%<strong>of</strong>thelogisapproximatedwitharelativeerrorbelow5%. Thisnetworkisoptimalfortheearthmodelsitwastrainedon.<br />

on,areacombination<strong>of</strong>thepreviousmodelswithoutandwithinvasion.A remarkshouldbemadethatthevalueRxowastakentobe1.1mwhen Thedomain<strong>of</strong>earthmodelstheconvolutional-regressionnetFMistrained 5.3Applicationtorealisticearthmodel<br />

fromarealoilwell.Thismodelismorerealistic.Itdoesnotlieintheinput dxowas4.25inch.InFigure48aformationmodelisshownthatcomes domain,becauseitcontainsbothpartswithandwithoutinvasion.Inour trainingseteachmodelonlycontainedeitherinvasionornot.Thebedsare relativelysmall.TheDeep-logandShallow-loglookverysimilarforthis donotexpectthenettobeoptimalforthismodel. thismodellooksquitedierentfromthemodelsinthetrainingsetandwe model,thisisalsodierentfromthemodelsthenetistrainedon.Allinall

74 5.THENEURALNETWORKASFASTFORWARDMODEL<br />

Adescription<strong>of</strong>theearthmodelsthenetwastrainedonisgiveninthe followingtable: net toolresponse sizeslidingwindow sizetrainingset rangeRt Deep-logandShallow-log 61688examples FM<br />

rangeRxo rangedxo 4.25,8...50inch 0.5...2.5m 1...71m 30feet<br />

rangebedsize speed accuracyShallow-log accuracyDeep-log 1...20feet 8.0% 9.0% 85pnt/sec<br />

imation<strong>of</strong>theShallow-logisshowninFigure50.Aswecanseefromthe Theapproximation<strong>of</strong>theDeep-logisshowninFigure49andtheapprox-<br />

cumulativeerrorplots,therelativeerrorbetweentheneuralnetresponse andtheDeep-logisbelow5%for67%<strong>of</strong>thelog.For88%<strong>of</strong>thelogthe relativeerrorliesbelow10%. TheShallow-logapproximationismuchbetter.Here77%<strong>of</strong>theloghasa relativeerrorbelow5%.

5.3Applicationtorealisticearthmodel 75<br />

100<br />

True resistivity Rt<br />

Invasion resistivity Rxo<br />

Invasion radius dxo<br />

10<br />

1<br />

200 210 220 230 240 250 260 270 280 290 300<br />

100<br />

True resistivity Rt<br />

Invasion resistivity Rxo<br />

Invasion radius dxo<br />

10<br />

Figure45:Examples<strong>of</strong>earthmodelsusedforthenetInvasion.<br />

1<br />

200 210 220 230 240 250 260 270 280 290 300

76 5.THENEURALNETWORKASFASTFORWARDMODEL<br />

Shallow-log<br />

Approximation by neural net<br />

100<br />

10<br />

1<br />

200 210 220 230 240 250 260 270 280 290 300<br />

Percentage <strong>of</strong> log<br />

with relative error<br />

below R %<br />

100<br />

90<br />

80<br />

70<br />

60<br />

50<br />

40<br />

30<br />

Figure46:Worstcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris7.5%.<br />

20<br />

10<br />

0<br />

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />

Relative error R

5.3Applicationtorealisticearthmodel 77<br />

Shallow-log<br />

Approximation by neural net<br />

100<br />

10<br />

1<br />

200 210 220 230 240 250 260 270 280 290 300<br />

Percentage <strong>of</strong> log<br />

with relative error<br />

below R %<br />

100<br />

90<br />

80<br />

70<br />

60<br />

50<br />

40<br />

30<br />

Figure47:Bestcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris4.5%.<br />

20<br />

10<br />

0<br />

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />

Relative error R

78 5.THENEURALNETWORKASFASTFORWARDMODEL<br />

True resistivity Rt<br />

Invasion resistivity Rxo<br />

Invasion radius dxo<br />

100<br />

Figure48:A(realistic)earthmodel.<br />

10<br />

1<br />

5400 5410 5420 5430 5440 5450 5460 5470 5480 5490 5500

5.3Applicationtorealisticearthmodel 79<br />

Deep-log<br />

Approximation by neural net<br />

100<br />

10<br />

1<br />

5400 5410 5420 5430 5440 5450 5460 5470 5480 5490 5500<br />

Percentage <strong>of</strong> log<br />

with relative error<br />

below R %<br />

100<br />

90<br />

80<br />

70<br />

60<br />

50<br />

40<br />

30<br />

Figure49:<strong>Neural</strong>netapproximation<strong>of</strong>Deep-log.Averagerelativeerroris 7.6%.<br />

20<br />

10<br />

0<br />

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />

Relative error R

80 5.THENEURALNETWORKASFASTFORWARDMODEL<br />

Shallow-log<br />

Approximation by neural net<br />

100<br />

10<br />

1<br />

5400 5410 5420 5430 5440 5450 5460 5470 5480 5490 5500<br />

Percentage <strong>of</strong> log<br />

with relative error<br />

below R %<br />

100<br />

90<br />

80<br />

70<br />

60<br />

50<br />

40<br />

30<br />

Figure50:<strong>Neural</strong>netapproximation<strong>of</strong>Shallow-log.Averagerelativeerror is8.3%.<br />

20<br />

10<br />

0<br />

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />

Relative error R

6Conclusions Inthissectionwepresenttheconclusionsfollowingfromtheexperimentswe haveperformedandthemethodswehaveinvestigated.Theconclusionsare splitintothreeparts 1.Analconclusionaboutthegoal<strong>of</strong>thisproject:investigatingwhether 2.Conclusionsabouttheusedmethods.Thisinvolvestheinputrepresentation,thetechniquesthatareusedtoreducethenumber<strong>of</strong>inputs<br />

itisfeasibletouseaneuralnetworkintheforwardmodellingprocess. andthearchitecturalconstraints. 3.Conclusionsabouttheuse<strong>of</strong>theneuralnetworkintrainingandtesting.Theseconclusionsareusefulinfurtherinvestigationandinthe<br />

Intherstpart<strong>of</strong>theprojectweonlyusedearthmodelswithoutinvasion 6.1<strong>Neural</strong>networkasfastforwardmodel? use<strong>of</strong>thetrainednetworkinotherapplications.<br />

types<strong>of</strong>earthmodelsneuralnetsweretrainedwiththefollowingresults: andinthesecondpartweonlyusedearthmodelswithinvasion.Forboth Noinvasion:Weusea\standard"fullyconnectednetwithonehiddenlayercontaining15nodes,aslidingwindow<strong>of</strong>15feetandasamplingperiod<strong>of</strong>0.2feet(75inputs).Thenetworkhasmoretrouble<br />

inlearningtheDeep-logthantheShallow-log.Thisiscausedbythe shoulderbedeect,whichismorepronouncedintheDeep-logthanin theShallow-log.Theperformanceisgiveninthefollowingtable TargetlogDataset Shallow-logTrainingsetwithrelativeerrorrelativeerror<br />

percentage<strong>of</strong>log below5%. 98% average<br />

Deep-log Testset 96% 62% 61% 1.7% 2.2% 5.1% 6.0%<br />

81

82 6.CONCLUSIONS<br />

Invasion:Wetrainedoneconvolutional-regressionnetontheShallowlog.Thenetusesaslidingwindow<strong>of</strong>29.8feet,sampledevery0.2feet,<br />

Theperformanceisgiveninthefollowingtable TargetlogDataset Shallow-logTrainingset withrelativeerrorrelativeerror percentage<strong>of</strong>log below5%. 88% average 27receptiveelds<strong>of</strong>3.8feetwith75%overlapand6featuremaps.<br />

Validationset 82% 4.3%<br />

Mixedinvasionandnoinvasion:Weusedthesameconvolutionalregressionnetasintheprevioustestswithatrainingsetconsisting<strong>of</strong><br />

6.2% Testset 77% 5.1%<br />

wetrainedthenetonboththeDeep-andtheShallow-log.Thesenets earthmodelswithandwithoutinvasion(butnotmixed).Thistime<br />

modelswithandwithoutinvasion Inthefollowingtabletheperformance<strong>of</strong>thenetsaregivenforthe arenotoptimal,becauserstthenetwasminimalizedonmodelswith invasionandthenweaddedmoremodelswithandwithoutinvasion. Targetlog Shallow-loginvasion Dataset relativeerror below5%. percentageaverage <strong>of</strong>logwithrelative 76% error<br />

noinvasionTrainingset Validationset 75% 77% 5.5%<br />

5.3%<br />

Deep-log 74% 52% 5.6%<br />

noinvasionTrainingset Validationset Testset 51% 50% 58% 56% 8.8% 9.0% 6.9% 8.4% 7.0%

6.2Methods 83<br />

Realloggingdata:Realearthmodelscontainbothlayerswithand earthmodel.Theperformance<strong>of</strong>thesenetsontherealloggingdata isTargetlog themixedinvasionandnoinvasionpartweretestedonarealistic withoutinvasion.Thetrainedconvolutional-regressionnetworksfrom<br />

Shallow-logwithrelativeerrorrelativeerror<br />

percentage<strong>of</strong>log below5%. 77% average<br />

Deep-log 67% 8.3%<br />

this(partlywithandpartlywithoutinvasion)andbychoosingthe Thenetworkperformancecanbeimprovedby<strong>using</strong>moremodelslike 7.6%<br />

Approximationtime:Thegoal<strong>of</strong>theprojectistocreateafaster thantheforwardmodelthatispresentlyusedatKSEPL(bothtimes forwardmodel.Theneuralnetworkisapproximately100timesfaster invasionradiusbetween4and25inchinstead<strong>of</strong>8and50inch.<br />

Althoughtheaccuracy<strong>of</strong>theapproximationstillneedssomeimprovement, thenetworkisalotfasterthantheforwardmodelthatisusednow.We measuredonanIBMR6000workstation).Thiscouldstillbeimproved<br />

concludethatitisfeasibletouseaneuralnetworkintheforwardmodelling byoptimization<strong>of</strong>theneuralnetcalculations.<br />

process.Evenwithlessaccuracytheneuralnetworkcouldbeusedinthe rstiterations<strong>of</strong>theforwardmodellingprocess.Inthatwayafairlygood initialguesscanbemadeveryquickly.Thenonecanusethemoreaccurate forwardmodel. 6.2Methods<br />

whereweusedearthmodelswithinvasion. Intherstpart<strong>of</strong>theprojectweonlyusedfullyconnectednetswithvarying window.Moreinterestingresultsarefoundinthesecondpart<strong>of</strong>theproject, number<strong>of</strong>layers,hiddennodesandvariationsinthesize<strong>of</strong>thesliding<br />

6.2.1Inputrepresentation Weexperimentedwithtwoinputrepresentation.Therst,thediscretized andhasbeenusedinfurthertests: slidingwindowapproach,gavethebestresults(trainingandgeneralization) Discretizedslidingwindow:Weusedauniformsampledsliding windowasinputtotheneuralnet.Thenumber<strong>of</strong>inputscomingfrom

84 6.CONCLUSIONS<br />

Attributes:Againweusedaxedsizeslidingwindow,butnowwe well. trainingresultwasgood,butthenetworkwasnotabletogeneralize thiswindowcanbequitehigh,especiallyinthecasewithinvasion.The<br />

describethebedsthatoccurinthiswindow.Eachbedwasdescribed byanumber<strong>of</strong>attributes.Theproblem,however,wasnotdescribed<br />

6.2.2Inputreduction task. wellbytheseattributes.Tondappropriateattributesisadicult<br />

Allpreprocessingmethodsweusedweresuccessful.Weachievedahigh inputreductionwithoutloss<strong>of</strong>performance(measuredintrainingandgeneralizationresults):<br />

Samplingmethod:Theslidingwindowcanbesampleduniformand plingperiods.Acoarsesamplingperiodresultsinlessinputsandthe non-uniform.Intherstcaseweexperimentedwithdierentsam-<br />

fromtheedges.Non-uniformsamplingresultsinlessinputsandthe receivesmost<strong>of</strong>itsinformationfromthecenter<strong>of</strong>thewindowandless ariesaredescribedlessaccurately.Thenon-uniformsamplingmethod workedverywell.Thismethodisbasedonthetoolphysics.Thetool trainingandgeneralizationresultswerecomparable.Thebedbound-<br />

Principalcomponents:Withtheprojection<strong>of</strong>theN-dimensional generalizationperformance<strong>of</strong>thenetwasbetterthanforthenetthat usedauniform-sampledslidingwindowasinput.Hereweonlyloose accuracyattheedges<strong>of</strong>thewindow.<br />

tion<strong>of</strong>theinputwaslost(measuredintherelativedistancebetween originaldiscretizedinput,althoughapproximately7%<strong>of</strong>theinforma-<br />

resultswerecomparableandthenetworkgeneralizedbetterthanthe components,aninputreduction<strong>of</strong>75%wasachieved.Thetraining inputvectortoaM-dimensionalvector,describedbyMprincipal<br />

Haartransform:Theadvantage<strong>of</strong>transformation<strong>of</strong>theoriginal cientscanberemoved.Areduction<strong>of</strong>73%<strong>of</strong>thenumber<strong>of</strong>inputs inputvaluestothewaveletcoecientsisthatanumber<strong>of</strong>thecoe-<br />

theoriginalandprojectedinput).<br />

wasachievedwithgoodtrainingandgeneralizationresults.Byremovingcoecientsattheedges<strong>of</strong>thewindow,weloosesomeaccuracy.<br />

lessaccurately. Thebedboundariesoutsidethecenter<strong>of</strong>thewindowaredescribed

6.3Application<strong>of</strong>theconvolutional-regressionnet 85<br />

6.2.3Architecturedesign Fullyconnectednets:Inthecasewithoutinvasionafullyconnected<br />

resultinginalongtrainingtimeandstorageandmemoryproblems. generalizewell.Thenumber<strong>of</strong>connectionsinthisnetisveryhigh, threeparameterstodescribethebed.Thistype<strong>of</strong>networkdoesnot netcanbeused.Thegeneralizationperformanceandthetraining<br />

Preprocessingmethodscanbeusedtoreducethenumber<strong>of</strong>inputs. resultsarecomparable.Whenthebedscontaininvasion,weneed<br />

Locallyconnectednets:Aneuralthatislocallyconnectedhas fullyconnectednets.Thesize<strong>of</strong>thereceptiveeldsandtheoverlap largeoverlap.Inthiswayitiseasierforthenettodeterminethe nets.Thetrainingtimeneededforthesenetsisshorterthanforthe arediculttodetermine.Wechoose,however,forsmalleldswith muchlessweightsandbettergeneralizationthanthefullyconnected<br />

Symmetryconstraints:Aninputsignalanditsmirrorimagegive preciselocation<strong>of</strong>afeaturethatoccursinone<strong>of</strong>theelds.Thisnet needsatleasttwohiddenlayers:onefordetectingthelocalfeatures andoneforcombiningthefoundfeatures. thesametoolresponseandshouldthereforealsogivethesamenetworkresponse.Thisrequirementleadstocertainweightconstraintduced,becausecertainweightsare\shared"(equal).Thereduction<br />

inthefully,locallyandwaveletnets.Thenumber<strong>of</strong>weightsisre-<br />

Convolutional-regressionnets:Intheconvolutional-regressionnet inthenumber<strong>of</strong>weightsmakesthetrainingmoredicult,butitimprovesthegeneralizationperformanceincomparisonwiththesame<br />

allreceptiveeldssharethesameweights.Onegroup<strong>of</strong>hiddennodes withsharedweightsiscalledafeaturemap.Thereduction<strong>of</strong>freedom netswithouttheseconstraints.<br />

set<strong>of</strong>sharedweights,thatishaslearneditself.Thegeneralization performance<strong>of</strong>thistype<strong>of</strong>netisbetterthanforalltheothernetswe iscompensatedbyanincreaseinthenumber<strong>of</strong>featuremaps.This netperformsaconvolutionontheinputwithaconvolutionkernel,the<br />

Whentheconvolutional-regressionnetisgoingtobeusedinotherapplications,thefollowingaspectsareimportant:<br />

Size<strong>of</strong>theslidingwindow:Thesize<strong>of</strong>theslidingwindowdepends ontheapplication.Oneshouldhaveanidea<strong>of</strong>howmuch<strong>of</strong>theinput<br />

6.3Application<strong>of</strong>theconvolutional-regressionnet havetried.<br />

logisresponsiblefortheoutput.

86 6.CONCLUSIONS<br />

Samplingperiod<strong>of</strong>slidingwindow:Thesamplingperiod<strong>of</strong>the<br />

Receptiveelds:Thesizeandoverlap<strong>of</strong>thereceptiveeldsdeterminetheaccuracy<strong>of</strong>thelocation<strong>of</strong>thefeaturesintheinput.Use<br />

goodindicationistousethesamesamplingperiodasforthetarget slidingwindowdetermineshowaccuratetheinputisdescribed.A<br />

smalleldswithhighoverlap.Thenumber<strong>of</strong>weights,however,dependonthenumber<strong>of</strong>receptiveelds.Choosethesize<strong>of</strong>theelds<br />

andtheoverlapsothatthenumber<strong>of</strong>weightsisnottoolarge. thetargetlogs.Donottakeatoodensesampling<strong>of</strong>thetargetlog, otherwisemostexampleslooktoomuchalike.Itisbettertotakea<br />

log.<br />

Representativetrainingset:Thetrainingsetisconstructedfrom<br />

Sizetrainingset:Thenumber<strong>of</strong>trainingexamplesthatareneeded (very)coarsesamplingperiodandusealargenumber<strong>of</strong>dierentlogs. examples.Createmodelsthatarelikelytobefoundinreality. Inthiswaythetrainingsetconsists<strong>of</strong>alargenumber<strong>of</strong>verydierent inordertogetgoodgeneralizationperformancedependsuponthe<br />

Minimalizationnetworkerror:Theerrorthatthenetworkisminimalizingcanbeadaptedtotherequirements<strong>of</strong>theapproximation.<br />

thenumber<strong>of</strong>weightsasexamples. ahighnumber<strong>of</strong>dierentexamples,oneneedsapproximately10times representativeness<strong>of</strong>thetrainingset.Whenthetrainingsetcontains<br />

Withthecombinedlogarithmicandnormalizationscalingtheneural<br />

morevariablesthanRt,Rxoanddxo.Whenthereareforexampledipping Theconvolutional-regressionnetcaneasilybeadaptedforproblemswith andaistheactualoutput. netisminimalizingtheproportiond=a,wheredisthedesiredoutput<br />

intheformation.Thescaling<strong>of</strong>anewvariableneedsnewinvestigation. Theboreholeradiusandresistivity<strong>of</strong>thedrillinguidcanalsobeadded asvariables,butlikethedipangle,thisshouldonlybedonewhenthereare point).Thisisonlypossiblewhenthereisadierentdipangleforeachbed layers,thedipanglecanbeaddedasfourthvariable(foreachsampling<br />

enoughdierentvalues.

A<strong>Neural</strong>networksimulators (vanCamp1993).Wealsotriedothernetworksimulators,Stuttgart<strong>Neural</strong> InthisprojectweusedtheXerionnetworksimulatorversions3.1and4.0 NetworkSimulator(SNNS)(Zell1993)andAspirin/MIGRAINES(Leighton designonanylevel:layers,nodesandevenconnections.Thisisveryuseful 1992),butwefoundthatXerionprovidesmorefreedominspecifyingarchitectureconstraints.TheXerionsimulatorallowsyoutoalterthenetwork<br />

fortheimplementation<strong>of</strong>theconvolutional-regressionnets,whereweconstraincertainweightstobeequal.<br />

optionfortimedelayedneuralnets(seeSection2.3.5).Alayerisspecied byitsnumber<strong>of</strong>features(orvariables)anditstotaldelaylength(number<strong>of</strong> TheSNNSsimulatorisabeautifulgraphicalsimulatorandhasaspecial<br />

Anditisalsonotpossibletochangetheoverlap<strong>of</strong>thereceptiveelds.We Itis,however,notpossibletoconstrainarbitraryweightstobethesame. eldsarespeciedbytheirsize(delaylength)andalwayshavemaximum overlap(displacement<strong>of</strong>onenode).Allreceptiveeldshavesharedweights. nodesinslidingwindow).Layerscanbefullyorlocallyconnected.Receptive<br />

presenttheinputsasasequencewithinaslidingwindow,butfortheSNNS<br />

periods.Thedelaylengthspecieshowmanyinputsbeforethisinputaect simulatoroneshouldpresenttheinputsatonespecicloggingpoint.The<br />

theoutput.Onecannotspecifyhowmanyinputsafterthisinputaectthe inourcase.Thereareonly4valuesperpattern(Rt,Rxo,dxoandthe advantage<strong>of</strong>thisisthatthelethatcontainsthedataismuchsmallerthan<br />

output.Sotherearethreemainpointswhythissimulatorisnotsuitable forourpurposes: toolresponse)instead<strong>of</strong>3(ws+1)forawindowsizewandasampling<br />

3.Onecanonlyspecifyhowmanyinputsbefore(andnotafter)the 2.Theoverlap<strong>of</strong>thereceptiveeldsisalwaysmaximal. 1.Itisnotpossibletospecifyarbitraryweightconstraints.<br />

Aspirin/MIGRAINESisacompiler.Itcompilesyourcodeintoaworkingneuralnetworksimulator(inC).Inthesourcecodetheneuralnetis<br />

Itisnotpossibletospecifyconvolutional-regressionnetswiththissimulator. currentinputaecttheoutput.<br />

layer,thesize<strong>of</strong>thelayer(innodes)andconnectioninformation(howthe speciedbyitscomponents.Acomponentisdescribedbythename<strong>of</strong>the calledSharedTessellationsandonecanspecifythesize<strong>of</strong>theeldsandthe overlapinthex-andy-direction.Itisnotpossible,however,toconstrain layerisconnectedtootherlayers).Receptiveeldswithsharedweightsare specicweightstobeshared,whichweneededinthesymmetryexperiments.<br />

I

II A.NEURALNETWORKSIMULATORS<br />

Itispossibletocreateconvolutional-regressionnetswiththissimulator. Althoughreceptiveeldsareeasierdenedbytheprevioussimulators,we haveusedtheXerionsimulator.Allhiddenneuronsinonefeaturemap areconnectedbynmlinkstotheircorrespondingreceptiveeld(<strong>of</strong>size nm).Alltheseconnectionshavetobespecied,whichcanmountupto 1620connectionsintheconvolutional-regressionnet.Neuronscanbeconstrainedtohavethesameincominglinks,whichiswhatwedidperfeature<br />

reading<strong>of</strong>thisleandthebuilding<strong>of</strong>thenetisquiteslow,butitoutweighs thelack<strong>of</strong>freedom<strong>of</strong>theothersimulators. weights.OurXeriontopologylescontainedatotal<strong>of</strong>9951lines.The map.Thenthenodesinonefeaturemapareconstrainedtohavethesame

REFERENCES<br />

References Anderson,B.&Barber,T.(1990).<strong>Modelling</strong>electromagnetictoolresponse,<br />

Chemali,R.,Gianzero,S.&Strickland,R.(1983).Theshoulderbedeffectontheduallaterologanditsvariationwiththeresistivity<strong>of</strong>the<br />

Asquith,G.(1982).Basicwellloganalysisforgeologists. OileldReviewpp.22{32.Welllogging.<br />

Chemali,R.,Gianzero,S.&Su,S.(1988).Theduallaterologincommon complexsituations,SPWLA29thannualloggingsymposium. boreholeuid,SPWLA24thannualloggingsymposium.<br />

Gianzero,S.(1977).Characteristicresponses<strong>of</strong>resistivitytoolsinelliptical Gianzero,S.,Lin,Y.&Su,S.(1985).Anewhigh-speedhybridtechnique boreholes,IEEEtransactionsongeoscienceelectronicsGE-15(4):251{ forsimulationandinversion<strong>of</strong>resistivitylogs. 256.<br />

Guyon,I.(1991).<strong>Neural</strong>networksandapplicationstutorial,PhysicsReports Haykin,S.(1994).<strong>Neural</strong><strong>Networks</strong>,AComprehensiveFoundation,MacmillanCollegePublisingCompany,Inc.<br />

(Reviewsection<strong>of</strong>PhysicsLetters)207(3{5):215{259.ReviewSection <strong>of</strong>PhysicalLetters207.<br />

Hertz,J.,Krogh,A.&Palmer,R.(1991).Introductiontothetheory<strong>of</strong> Leighton,R.R.(1992).TheAspirin/MIGRAINES<strong>Neural</strong>NetworkS<strong>of</strong>tware,v6.0edn.<br />

neuralcomputation,Addison-WesleyPublishingCompany.<br />

Strang,G.(1989).Waveletsanddilationequations:Abriefintroduction, Moran,J.(1985).Focusedresistivitylogs,Developmentsingeophysical SIAMReview31(4). explorationmethods6:225{260.<br />

vanCamp,D.(1993).AUsersGuideforTheXerion<strong>Neural</strong>NetworkSimulator,ver3.1&ver4.0edn,University<strong>of</strong>Toronto.<br />

Waibel,A.,Hanazawa,T.,Hinton,G.,Shikano,K.&Lang,K.(1989). Phonerecognition<strong>using</strong>time-delayneuralnetworks,IEEETransactionsonacoustics,speechandsignalprocessing37(3).<br />

Zell,A.(1993).SNNSStuttgart<strong>Neural</strong>NetworkSimulator,v3.1edn,InstituteforParallelandDistributedHighPerformanceSystems.<br />

III

logo

products

Resources

Company

Terms of service
Privacy policy
Cookie policy
Cookie settings
Imprint
Change language
Made with love in Switzerland
© 2024 Yumpu.com all rights reserved

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!