Forward Modelling of Resistivity Logs using Neural Networks - Liacs
Forward Modelling of Resistivity Logs using Neural Networks - Liacs
Forward Modelling of Resistivity Logs using Neural Networks - Liacs
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
<strong>Forward</strong><strong>Modelling</strong><strong>of</strong><strong>Resistivity</strong><strong>Logs</strong> <strong>using</strong><strong>Neural</strong><strong>Networks</strong> August31,1995 P.Demmenie
ation<strong>of</strong>rocks.Tothisend,ameasuringdeviceisloweredinawellboreto Abstract<br />
recordaso-calledresistivitylog.Duetoenvironmentaleectsthisresistivitylogdiersfromtheresistivity<strong>of</strong>theformation,thetrueresistivity.To<br />
hydrocarbons(oilandgas)andwatertodeterminethehydrocarbonsatu-<br />
Intheoilindustry,oneusesthedierenceinelectricalconductivitybetween<br />
invertthemeasuredlogtothetrueresistivityoneusesaniterativeforward modellingprocess,involvingthenumericalsolution<strong>of</strong>dierentialequations. Althoughthecurrentmodellingalgorithmshavesignicantlyimprovedin<br />
Theone-waymappingbetweentheearthmodels(trueresistivitymodeland neuralnetworksareveryfastinproducingoutputtocertaininput. siononawellsite.Therefore,wehaveinvestigatedthefeasibility<strong>of</strong><strong>using</strong> neuralnetworkstoperformtheforwardmodellingprocess.Oncetrained, speedcomparingtoafewyearsago,theyarestillnotfastenoughforinver-<br />
net(6750).Thegeneralizationperformance<strong>of</strong>thesenetsisnotsucient number<strong>of</strong>connectionsbetweentheinputlayerandthehiddenlayer<strong>of</strong>the <strong>of</strong>inputsthatisneededtorepresenttheearthmodel(450)andthehigh dard"fullyconnectednet.However,problemsarisefromthehighnumber environmentalconditions)andthetoolresponsecanbelearnedbya\stan-<br />
forthepurpose<strong>of</strong>resistivityloginversion.<br />
<strong>of</strong>fullyconnectednets.Wehavefoundanarchitecturewhichprovestobe number<strong>of</strong>connectionsinthenetby<strong>using</strong>locallyconnectednetsinstead principalcomponentanalysisandthewavelettransform.Thispreprocessing <strong>of</strong>theinputisquitesuccessful.Wehavealsostudiedmethodstoreducethe Wehavestudiedtwopreprocessingmethodstoreducethenumber<strong>of</strong>inputs:<br />
thesesharedweightsthehiddenlayerperformsaconvolution<strong>of</strong>theinput <strong>of</strong>netisbasedonthelocallyconnectednetsandsharedweights.Dueto quiteecientforthismapping:aconvolutional-regressionnet.Thistype layer.Theconvolutionkernel,theset<strong>of</strong>sharedweights,islearnedbythe netitself.Theweightsharingreducesthenumber<strong>of</strong>connectionsinthenet forwardmodelthatisusedatKSEPLnowadays.Theperformance<strong>of</strong>thenet ismeasuredintheaveragerelativeerrorbetweenthenetworkoututandthe andtheseconstraintsimprovethegeneralizationability<strong>of</strong>thenet. Thisconvolutional-regressionnetisapproximately100timesfasterthanthe whentheaveragerelativeerrorliesbelow5%.Theconvolutional-regression forwardmodeloutput.Theneuralnetcanbeusedasafastforwardmodel, netachievesanaccuracy<strong>of</strong>approximately8%onaresistivitylogcoming froma\real"oilwell.Furtherimprovementsinaccuracycanbeachieved by<strong>using</strong>amorerepresentativetrainingset.Evenwithlessaccuracythe neuralnetcouldbeusedasinitialstartfortheforwardmodellingprocess.<br />
i
Preface Thisreportisthemasterthesis<strong>of</strong>PamelaDemmenieforgraduatingfrom theRijksUniversiteitLeiden(RUL)atthedepartment<strong>of</strong>ComputerScience. andDr.IdaSprinkhuizen-Kuyper(RUL). ItdescribesaprojectthatwasperformedattheKoninklijkeShellExploratie enProduktieLaboratorium(KSEPL)inRijswijkfromDecember1994until August1995.TheprojectwassupervisedbyDr.GuozhongAn(KSEPL)<br />
Chapter3and4describetheactualexperimentsfordatawithoutandwith groundonforwardmodellingandashortintroductioninneuralnetworks. Thereportisdividedintosixchapters.Therstchapterprovidesaback-<br />
invasion.Inchapter5wepresenttheresults<strong>of</strong>thetrainedneuralnetworks Thesecondchapterdescribesallthemethodswehaveusedintheexperiments.Thesemethodsinvolveinputreductionsandarchitectureconstraints.<br />
onrealisticloggingdataandinchapter6wesummarizetheconclusions<strong>of</strong> thisproject. AllneuralnetsimulationswererunonSparc10andSparc20(Unix)stationsandshouldeventuallyberunonanIBMR600workstation(theapproximationtimesweremeasuredonthistype<strong>of</strong>workstation).Weused<br />
withothersimulators:Stuttgart<strong>Neural</strong>NetworkSimulator(SNNS)and Aspirin/MIGRAINES.Theadvantagesanddisadvantages<strong>of</strong>thesenetwork theXerionsimulator,versions3.1and4.0.Wehavealsoexperimented<br />
Sprinkhuizen-Kuyperfortheirsupportandideasduringthisproject.Of Acknowledgements First<strong>of</strong>allIwouldliketothankmysupervisorsGuozhongAnandIda simulatorsareoutlinedinappendixA.<br />
vanDijkandLeonHoman,whohaveprovidedthedatathatwehavebeen coursetheprojectwouldnothavebeencompletedwithoutthehelp<strong>of</strong>Niels workingwithandhelpedmeinmyunderstanding<strong>of</strong>theforwardmodelling versityfortheirsupportandcompanyduringtheninemonthsIhavebeen process.IalsoliketothankthestudentsfromKSEPLandfromtheuni-<br />
PimBussinkforword2onpage42. workingontheproject.Andlastbutnotleasteverlastinggratitudegoto<br />
ii
Contents Abstract Preface 1<strong>Forward</strong><strong>Modelling</strong><strong>of</strong><strong>Resistivity</strong><strong>Logs</strong> ii 1i<br />
1.1<strong>Resistivity</strong>Logging::::::::::::::::::::::::1 1.2Inversionby<strong>Forward</strong><strong>Modelling</strong>:::::::::::::::::5 1.2.1Inversion:::::::::::::::::::::::::5 1.1.1Loggingtools:::::::::::::::::::::::1<br />
1.2.2<strong>Forward</strong><strong>Modelling</strong>::::::::::::::::::::6 1.1.2Environmentaleects::::::::::::::::::3<br />
1.3<strong>Neural</strong><strong>Networks</strong>:::::::::::::::::::::::::7 1.3.3Thetrainingset:::::::::::::::::::::10 1.3.1Networkarchitecture:::::::::::::::::::9 1.3.4Generalization::::::::::::::::::::::11 1.3.2TheLearningMethod::::::::::::::::::9<br />
2Inputrepresentationandarchitecturedesign 1.4<strong>Forward</strong><strong>Modelling</strong><strong>using</strong>a<strong>Neural</strong>Network::::::::::12 2.1Inputrepresentation:::::::::::::::::::::::15 1.3.5Localminima:::::::::::::::::::::::12<br />
2.1.1Discretizedslidingwindow::::::::::::::::16 2.1.2Attributes:::::::::::::::::::::::::18 15<br />
2.2Preprocessing:::::::::::::::::::::::::::20 2.1.3Inputandoutputscaling::::::::::::::::19<br />
2.3Architectureconstraints:::::::::::::::::::::27 2.3.1Fullyconnectednets:::::::::::::::::::27 2.2.1PrincipalComponentAnalysis:::::::::::::20<br />
2.3.2Locallyconnectednets::::::::::::::::::28 2.2.2Wavelettransform::::::::::::::::::::22<br />
2.3.6Convolutional-regressionnetwork::::::::::::34 2.3.5TimeDelayednetwork::::::::::::::::::33 2.3.4Convolutionalnetwork::::::::::::::::::32 2.3.3Symmetryconstraints::::::::::::::::::28<br />
3<strong>Forward</strong>modellingwithoutmudinvasion 2.4Errorfunction:::::::::::::::::::::::::::35 3.2Experimentingwiththenetworkarchitecture:::::::::38 3.3Experimentingwiththesize<strong>of</strong>theslidingwindow::::::39 3.1Experimentingwithdierentscalingmethods:::::::::38 37<br />
3.4Summaryandresults:::::::::::::::::::::::39
4<strong>Forward</strong>modellingwithmudinvasion 4.3Experimentingwithinputreductionmethods:::::::::46 4.2Experimentingwithdierentinputrepresentations::::::44 4.1Scaling<strong>of</strong>theparameters::::::::::::::::::::43 4.3.1Usingdierentsamplingmethods::::::::::::46 42<br />
4.4Creatingamorerepresentativetrainingset::::::::::53 4.3.3Reducingtheinputsbyremovingwaveletcoecients:50 4.3.2Reducingtheinputbyprojectiontoprincipalcomponents::::::::::::::::::::::::::::47<br />
4.6Experimentingwitharchitectureconstraints::::::::::57 4.5Intermediateresultsinputrepresentations:::::::::::55<br />
4.7Intermediateresultsarchitecturedesign::::::::::::65 4.6.4Experimentingwithconvolutionalregressionnets:::59 4.6.3Usingsymmetryconstraints:::::::::::::::57 4.6.2Experimentingwithlocallyconnectednets:::::::57 4.6.1Experimentingwithfullyconnectednets::::::::57<br />
5Theneuralnetworkasfastforwardmodel 5.2Applicationtoearthmodelswithinvasion:::::::::::73 5.1Applicationtoearthmodelswithoutinvasion:::::::::68 4.8Summaryandresults:::::::::::::::::::::::65<br />
5.3Applicationtorealisticearthmodel:::::::::::::::73<br />
6Conclusions 6.2Methods::::::::::::::::::::::::::::::83 6.1<strong>Neural</strong>networkasfastforwardmodel?:::::::::::::81 6.2.1Inputrepresentation:::::::::::::::::::83 6.2.2Inputreduction::::::::::::::::::::::84<br />
A<strong>Neural</strong>networksimulators 6.3Application<strong>of</strong>theconvolutional-regressionnet::::::::85 6.2.3Architecturedesign::::::::::::::::::::85I
List<strong>of</strong>Tables<br />
54231 Parametersforearthmodelswithinvasion.::::::::::42 Performanceonearthmodelswithoutinvasion(1).::::::40 Parametersforearthmodelswithoutinvasion.::::::::37 Averagerelativeerrorforearthmodelswithoutinvasion(1).39 Thetoolresponse.::::::::::::::::::::::::13<br />
876 plingperiods.:::::::::::::::::::::::::::45 Comparinguniformsamplingmethodswithdierentsam-<br />
Comparingdiscretizedslidingwindowagainstattributesrepresentation.::::::::::::::::::::::::::::45<br />
9 Comparinguniformandnon-uniformsamplingmethods.::45 Comparinguniformsampledinputwithoutandwithprojectiontoprincipalcomponents.::::::::::::::::::48<br />
10Comparingnon-uniformsampledinputwithoutandwithprojectiontoprincipalcomponents.::::::::::::::::49<br />
11Comparinginputrepresentationswithdierentnumber<strong>of</strong> 12Comparinginputrepresentationswithdierentnumber<strong>of</strong> 13Comparingtrainingset<strong>of</strong>targetlogsandtrainingset<strong>of</strong> waveletcoecients(1).::::::::::::::::::::::51 waveletcoecients(2).::::::::::::::::::::::51<br />
16Comparingdierentlocallyconnectednets.::::::::::58 15Comparingdierentfullyconnectednets.:::::::::::58 14Comparingtrainingset<strong>of</strong>targetlogsandtrainingset<strong>of</strong>dif- cultparts<strong>of</strong>targetlogs.::::::::::::::::::::54 coarsersampledtargetlogs.:::::::::::::::::::54<br />
19Comparing\wavelet"netwithoutandwithsymmetryconstraints.::::::::::::::::::::::::::::::6tryconstraints.::::::::::::::::::::::::::60<br />
constraints.::::::::::::::::::::::::::::58 18Comparinglocallyconnectednetswithoutandwithsymme-<br />
17Comparingfullyconnectednetswithoutandwithsymmetry<br />
20Resultsforconvolutional-regressionnets(1).:::::::::63 21Resultsforconvolutional-regressionnets(2).:::::::::63 22Averagerelativeerrorandperformanceforearthmodelswith 23Averagerelativeerrorandperformanceforearthmodelswith 24Averagerelativeerrorandperformanceforearthmodelswith invasion(Shallow-log).::::::::::::::::::::::65 andwithoutinvasion(Shallow-log).:::::::::::::::66 andwithoutinvasion(Deep-log).::::::::::::::::67
List<strong>of</strong>Figures 132 Laboratoryresistivitymeasuringapparatuswithunguarded planarelectrodes.:::::::::::::::::::::::::1<br />
part<strong>of</strong>theformation.::::::::::::::::::::::3 InputtotheDeepLaterolog(left)andtheShallowLaterolog (right).TheDeepLaterologgetsinformationfromalarger electrodesinstead<strong>of</strong>planarelectrodesandtheDual-Laterolog.2 Modication<strong>of</strong>thelaboratoryapparatus<strong>using</strong>cylindrical<br />
456 Environmentaleects:(A)idealsituation,(B)dippingformation,(C)cavedboreholes,(D)deviatedboreholesand<br />
Theinversionprocessandforwardmodelling<strong>of</strong>resistivitylogs.6 (E)horizontalboreholes.::::::::::::::::::::4<br />
10Part<strong>of</strong>amodelwithinputsignalRtandoutputsignalsDeeplog(LLd)andShallow-log(LLs).TheinputsignalsRxoand<br />
879 Localminimaintheerrorsurface.:::::::::::::::12 Adiagram<strong>of</strong>neuronj.::::::::::::::::::::::8 Aformationanditsmodel.:::::::::::::::::::13 Transferimpedances.:::::::::::::::::::::::6<br />
12Whentheslidingwindowissmallerthanthelargestbedinthe 11Example<strong>of</strong>slidingwindowinputrepresentation.Asliding dxoareomitted.:::::::::::::::::::::::::14<br />
situation,althoughthetargetdiersalot.:::::::::::17 model,theinputtothenetwillbethesameforthesketched theinputmodels.:::::::::::::::::::::::::15 window<strong>of</strong>sizewisplacedaroundthepoint<strong>of</strong>interestalong<br />
16Theinputsignaliswrittenasaweightedcombination<strong>of</strong>rectangularblockfunctions.Thecoecientsciareusedasinputs<br />
totheneuralnet.:::::::::::::::::::::::::25<br />
13Standardnormaldistribution.::::::::::::::::::19 15HaarwaveletfromboxfunctionWHaar(x)=(2x)(2x1).23 14Most<strong>of</strong>thevariation<strong>of</strong>thedataliesinthedirection<strong>of</strong>1.:21<br />
17Boundariesoutsidethecenter<strong>of</strong>thewindowarenotdescribed 20Adiscretizedinputsignalanditsmirrorimage.::::::::29 21Symmetryconstraintsonfullyconnectednet.:::::::::30 18Fullyconnected(left)andlocallyconnected(right)neuralnets.28 19Receptiveeldswhichoverlap13.::::::::::::::::29 accurately.:::::::::::::::::::::::::::::26<br />
23Symmetryconstraintsonwaveletnets.Thecoecientsare 22Symmetryconstraintsonlocallyconnectednet.Thereceptiveeldsareconstrainedsymmetrically(receptiveeldfis<br />
24Convolutionalnetwork.::::::::::::::::::::::32 constrainedperdetaillevel.::::::::::::::::::31 constrainedtoeldFf+1forFelds).::::::::::30
29Response<strong>of</strong>theneuralnettothedicultmodelshownabove.41 28ModelthatcausesdicultiesinapproximatingtheDeep-log.41 27Thenetworkandrelativeerrorforxedproportionsd=a.::35 26Convolutional-regressionnetwork.::::::::::::::::34 25Timedelayedneuralnetwork.::::::::::::::::::33<br />
31Non-uniformsamplinginslidingwindow.:::::::::::47 30Smalldierenceininputwithattributesinputrepresentation.44<br />
33Loss<strong>of</strong>informationpervariableforTest7.Originalinput 32Loss<strong>of</strong>informationpervariableforTest6.Originalinput consists<strong>of</strong>364inputs,comingfromanon-uniformsampled slidingwindow.::::::::::::::::::::::::::48 slidingwindow.::::::::::::::::::::::::::49 consists<strong>of</strong>3128inputs,comingfromauniformsampled<br />
35Intermediateresults<strong>of</strong>trainingthreedierentneuralnetson 34Notalltransitionsaredetectedbywaveletcoecients.:::52<br />
37Convolutional-regressionnet.Thersthiddenlayerconsists 36Convolutional-regressionnet.Thefeaturemapsintherst 6models.:::::::::::::::::::::::::::::56<br />
<strong>of</strong>threesets<strong>of</strong>featuremaps.Eachsetconsists<strong>of</strong>threemaps andisconnectedtoone<strong>of</strong>thevariablesintheinputlayer.:61 layer.:::::::::::::::::::::::::::::::59 hiddenlayerareconnectedtoallthevariablesintheinput<br />
39Activation<strong>of</strong>rsthiddenlayerperfeaturemapforTest21. 38Trueresistivityprole<strong>of</strong>modelAatdepth615feet.:::::64<br />
41Worstcaseneuralnetapproximation<strong>of</strong>Deep-log.Average 40Intermediateresults(2)<strong>of</strong>trainingdierentneuralnetson6 models.::::::::::::::::::::::::::::::66 TheseactivationsareformodelAatdepth615feet.This<br />
relativeerroris14.2%.:::::::::::::::::::::69 layerconsists<strong>of</strong>6featuremaps<strong>of</strong>27nodeseach.:::::::64<br />
42Bestcaseneuralnetapproximation<strong>of</strong>Deep-log.Average 43Worstcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average 44Bestcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris2.6%.::::::::::::::::::::::70<br />
45Examples<strong>of</strong>earthmodelsusedforthenetInvasion.:::::75 46Worstcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris8.5%.::::::::::::::::::::::71<br />
47Bestcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris1.2%.::::::::::::::::::::::72<br />
49<strong>Neural</strong>netapproximation<strong>of</strong>Deep-log.Averagerelativeerror 48A(realistic)earthmodel.::::::::::::::::::::78 relativeerroris7.5%.::::::::::::::::::::::76 relativeerroris4.5%.::::::::::::::::::::::77 is7.6%.:::::::::::::::::::::::::::::79
50<strong>Neural</strong>netapproximation<strong>of</strong>Shallow-log.Averagerelative erroris8.3%.:::::::::::::::::::::::::::80
1<strong>Forward</strong><strong>Modelling</strong><strong>of</strong><strong>Resistivity</strong><strong>Logs</strong><br />
andhydrocarbonsdonotconductelectriccurrents,whereastheformation Therockporescanbelledwithwaterorhydrocarbons(oilandgas).Rock 1.1<strong>Resistivity</strong>Logging<br />
hydrocarbonsaturationwithintherocks.Thehydrocarbonsaturationisan waterdoes.Onereasonformeasuringtheresistivityistodeterminethe Aformationconsists<strong>of</strong>severallayers(beds)<strong>of</strong>rock,whichcontainpores.<br />
indicationforthepresence<strong>of</strong>oil.Asimpliedexpression<strong>of</strong>thisquantitative aspectisexempliedbyArchie'sequation<br />
HereRwistheresistivity<strong>of</strong>thewaterintherockpores,Fistheformation Sw=FnRw<br />
factorgenerallyassumedtobederivablefromaknowledge<strong>of</strong>therockresistivity,Swisthewatersaturation(percentage<strong>of</strong>porespaceoccupiedby<br />
water),Rtisthemeasuredrockresistivityandnisanempiricallydeter-<br />
(Moran1985). 1.1.1Loggingtools minedsaturationexponent.HydrocarbonsaturationShisequalto1Sw<br />
Rt (1)<br />
Most<strong>of</strong>thephysicsbehindtheresistivityloggingtechniquescanbefound in(Moran1985).Inthissectionwewilldiscussthebasicideasbehindthese techniques.<br />
1<br />
electrode B<br />
V<br />
electrodes. Asheet<strong>of</strong>materialwhoseresistivityistobedeterminedisplacedbetween Figure1:Laboratoryresistivitymeasuringapparatuswithunguardedplanar andincontactwithelectrodesAandB,both<strong>of</strong>areaSasisshownin<br />
area S<br />
A<br />
distance dr in meter
electrode A<br />
I<br />
0<br />
I<br />
1<br />
I<br />
2<br />
I<br />
1<br />
2 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />
(m)yields Figure1.AvoltageVisappliedtotheelectrodes,resultinginacurrent application<strong>of</strong>Ohm'slawformaterialthicknessdr(m)andresistivity I.IfIisdistributeduniformlyovertheareaSandiszerooutside,an<br />
Theresistivityisfoundbymeasuringthevoltagedropbetweentheelectrodes.Inpractice,thecurrentIwillnotbeuniformlydistributedoverthe<br />
discs,butwilltendtobemaximumattheedgesandminimuminthecentre. V=ISdr (2)<br />
disc<strong>of</strong>areaS0carryingacurrentI0surroundedbytheremainder<strong>of</strong>the Toimproveonthisscheme,onesplitsthediscsothatonehasasmallcentral discsheldatthesamepotentialVasS0.Thiswillresultinnearlyconstant currentdensityoverS0.ThemeasuredcurrentI0willbe\focused".The resistivityisnowgivenmoreaccuratelythanbeforeby =S0 drV I0 (3)<br />
A1<br />
I<br />
0<br />
I<br />
0<br />
A0<br />
I<br />
Figure2:Modication<strong>of</strong>thelaboratoryapparatus<strong>using</strong>cylindricalelectrodesinstead<strong>of</strong>planarelectrodesandtheDual-Laterolog.<br />
D D0<br />
1<br />
A2<br />
I0<br />
I1<br />
I<br />
2<br />
I<br />
Thisideaisthebasicingredient<strong>of</strong>theLaterolog.TherealLaterologuses<br />
1<br />
cylindricalelectrodesinstead<strong>of</strong>planarelectrodesasindicatedinFigure2<br />
I<br />
2<br />
B<br />
aDeepLaterolog.TheDeepLaterologhasitscurrentreturnelectrodeB (left).TheDualLaterologconsists<strong>of</strong>aShallow(Pseudo)Laterologand<br />
SHALLOW DEEP<br />
remotelyatthesurface,whichresultsincurrentsasshowninFigure3.The
1.1<strong>Resistivity</strong>Logging 3<br />
DeepLaterologreadsfarintotheformation.TheShallowLaterologhas itscurrentreturnelectrodeplacedaboveandbelowtheelectrodeA,which makesthecurrentsbendbacktothetoolasshowninFigure3.ThisLaterologreadstheformationclosetothetool,whichmakesitmoresensitive<br />
totheinvadedzone(seeSection1.1.2). Bothmeasurementsaremadesimultaneouslyby<strong>using</strong>dierentfrequencies; quencyfortheDeepLaterolog. arelativelyhighfrequencyfortheShallow-Laterologandaverylowfre-<br />
part <strong>of</strong> the<br />
formation that<br />
is seen by the<br />
tool<br />
part <strong>of</strong> the<br />
formation that<br />
is seen by the<br />
tool<br />
formation. Figure3:InputtotheDeepLaterolog(left)andtheShallowLaterolog<br />
tool<br />
tool<br />
1.1.2Environmentaleects (right).TheDeepLaterologgetsinformationfromalargerpart<strong>of</strong>the<br />
Severalanomalies,likeinvasion,dippingbeds,washed-outboreholesand<br />
in(Gianzero1977),(Asquith1982),(Chemali,Gianzero&Strickland1983) occurareshowninFigure4andmoredetaileddescriptionscanbefound <strong>of</strong>theformation.Theseanomalieshaveaneectonthetoolresponse, whicharediculttomodel.Situationsinwhichtheseenviromentaleects veryresistiveformations,areencounteredwhenmeasuringtheresistivity<br />
pressure,whichpreventsblow-outs. and(Chemali,Gianzero&Su1988). Wellsaredrilledwithrotarybits.Specialdrillingmudisusedascirculating uid.Themudremovescuttingsfromthewellbore,lubricatesandcoolsthe drillbitandhelpsmaintaininganexcess<strong>of</strong>boreholepressureoverformation
4 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />
Thispressuredierenceforcessome<strong>of</strong>thedrillinguidtoinvadeporous andpermeableformation.Inthisprocess<strong>of</strong>invasionsolidparticles(clay mineralsfromthedrillingmud)aretrappedontheside<strong>of</strong>theboreholeand formaso-calledmudcake.Thepart<strong>of</strong>theformationwhichisinvadedby mudltrateiscalledtheinvadedzone.<br />
A<br />
B<br />
tool<br />
current flow<br />
C D E<br />
Thesize<strong>of</strong>theboreholeandthemudcakeresistivityinuencethemeasured (C)cavedboreholes,(D)deviatedboreholesand(E)horizontalboreholes. Figure4:Environmentaleects:(A)idealsituation,(B)dippingformation,<br />
eectiveinmoderatelysalinetoverysalinemud,andlesseectiveinfresh resistivity.Thesmoothing<strong>of</strong>anomaliesonthelogbytheboreholeisquite IndippingformationstheDualLaterolog-curvesvaryslowlyacrossbed portionsandtheerrorduetoshoulderbedeect(theinuence<strong>of</strong>thebeds mud.Thesize<strong>of</strong>theboreholeandtheresistivity<strong>of</strong>thedrillinguidare<br />
adjacenttothecurrentbed)isdierentfromthenon-dippingcase. boundaries.Theapparentbedthicknessisincreasedinpredictablepro-<br />
takenxedinourstudy.<br />
tweenbedsalsoaectthetoolreadings.Formationslikelimestoneand occuronlyatabruptchangesinholediameter. Veryresistiveformationsandformationswithhighresistivitycontrastsbe-<br />
<strong>of</strong>theincreasedholediameter.Anomalousreadings<strong>of</strong>theDeepLaterolog Incavedboreholes(wash-out)theShallowLaterologissensitivetotheeect<br />
dolomitecanhaveresistivitiesover2000m,whilemostformationshavea resistivitybetween1and70m. OnlytheShallowLaterologexhibitssomesensitivitytoeccenteringinlarge boreholes. Thetoolresponseisessentiallyimmunetoasmalleccentering<strong>of</strong>thetool.
1.2Inversionby<strong>Forward</strong><strong>Modelling</strong> 5<br />
Theoveralleect<strong>of</strong>anellipticalboreholeisthatitproducescharacteristicresponseswhichliebetweenthoseobtainedintwocircularholeswith<br />
diametersequaltothemajorandminoraxes<strong>of</strong>theellipticalhole. importantinconductivemudthanitisinnon-conductivemud.TheShallowLaterologhasmuchlessshoulderbedeectthantheDeepLaterolog,<br />
Deviatedboreholesandhorizontalboreholesalsogivedierenttoolreadings.Comparethecurrentowintheidealcase(Figure4)andthesetypes<br />
<strong>of</strong>boreholes. 1.2Inversionby<strong>Forward</strong><strong>Modelling</strong> 1.2.1Inversion Thefoundresistivitylogs,Shallow-andDeep-log,havetobeinvertedto thetrueresistivity<strong>of</strong>theformation.Thisisdonebyaniterativeforward rstcorrectedwithchartbookcorrections(tornadochartsforexample;for boreholesize,invasion,etc.)Thenaninitialguessfortheformationmodelis modellingprocess,whichissketchedinFigure5.Theactualeldlogsare<br />
TheshoulderbedcorrectionrequiredfortheDeepLaterologismuchless especiallyforbedthicknessesabovetenfeet.<br />
boreholediameterandbeddingdip,thicknessandresistivity(Anderson& geometryand\parametervalues"|numbersassignedtovariablessuchas Barber1990).Thenthetoolphysicsisusedtocomputeanexpectedlog, made.Thistrialmodelincludesadescription<strong>of</strong>theboreholeandformation whichiscomparedwiththeactualeldlog.Ifthematchisnotgoodenough,<br />
<strong>Forward</strong><strong>Modelling</strong>)andamatchingprocedure.Thelatterprocedureimplies guessedformationmodeltotheexpectedDeep-andShallow-log(so-called Theprocessconsists<strong>of</strong>twosteps:thefunctionapproximationfromthe isiterateduntilthetwologsmatchsatisfactorily. theinitialtrialmodelisalteredandthecalculationrepeated.Thisprocess<br />
aminimalization<strong>of</strong>amatchingerrorbetweenthetwologs.Thiscanbe maximumentropymethods(Anderson&Barber1990). donebyhandor<strong>using</strong>sophisticatedalgorithmsliketheleast-squaresand
6 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />
Formation model<br />
Expected log<br />
<strong>Forward</strong><br />
<strong>Modelling</strong><br />
Adapt<br />
Matching<br />
initial guess<br />
<strong>of</strong> formation<br />
procedure<br />
1.2.2<strong>Forward</strong><strong>Modelling</strong> Figure5:Theinversionprocessandforwardmodelling<strong>of</strong>resistivitylogs.<br />
<strong>of</strong>theprocess.TocomputetheLaterologresponseforagivenelectrodearray Theforwardmodellingpart<strong>of</strong>theinversionisthemosttime-consumingpart<br />
Actual measured field V1 I1<br />
Z 12<br />
Z<br />
11<br />
Z<br />
V<br />
01<br />
Z<br />
0 I<br />
00<br />
0<br />
Z<br />
02<br />
anddistribution<strong>of</strong>resistivities,itissucienttodeterminetheassociated<br />
Z<br />
transferimpedances.ThetransferimpedanceZijisequaltoVj<br />
V2<br />
I<br />
22<br />
2<br />
thevoltagemeasuredatpartj<strong>of</strong>theelectrodecongurationandIiisthe Figure6:Transferimpedances.<br />
currentemittedbyparti.Aspecicelectrodecongurationanditstransfer Ii,whereVjis<br />
ELECTRODE<br />
TRANSFER<br />
CONFIGURATION IMPEDANCES
1.3<strong>Neural</strong><strong>Networks</strong> 7<br />
insimpliedmodels(concentriccylindricalboundariesorplaneboundaries) impedancesareshowninFigure6.Forthecomputation<strong>of</strong>theresponse onecanuseanalyticalapproaches,butwithextendedelectrodesforexample,<br />
certainboundaryconditions. Zijinvolvesthesolution<strong>of</strong>Laplace'sequationintwospacevariablesunder beenpublished(Moran1985).Thedetermination<strong>of</strong>thetransferimpedances dippinglayers,theproblemis<strong>of</strong>suchcomplexitythatnoresultshaveasyet theproblemcanonlybehandledbynumericalmethods.Inthecase<strong>of</strong><br />
Thereareanumber<strong>of</strong>methodsthatcanbeusedtosolvetheseboundary valueproblems,wementiontheFiniteElementMethod,theBoundaryelementornite-dierencetechniqueandaHybridMethod(Gianzero,Lin&<br />
Su1985).<br />
(horizontal)dependenceistreatednumericallyandtheaxial(vertical)dependenceanalytically.Itcombinesthemodeconceptinwave-guidetheory<br />
withtheFiniteElementMethod.Thismethod,employedbyGianzero,has ElementMethod(Gianzeroetal.1985). beenabletosimulatea100footlogwith25bedsinlessthantwelveminutesonanIBM3081,whichisapproximately8timesfasterthantheFinite<br />
directconsequence<strong>of</strong>minimizingthetotalenergy<strong>of</strong>thesystem. ciple(Chemalietal.1983).ItcanbeshownthatLaplace'sequationisa TheHybridMethodisaseparation<strong>of</strong>variablesapproachwheretheradial TheFiniteElementMethodisanumericalmethodbasedonanenergyprin-<br />
A<strong>Neural</strong>Networkconsists<strong>of</strong>anumber<strong>of</strong>layersthatconsist<strong>of</strong>nodes(neurons).Aneuronreceivesinputfromtheneuronsitisconnectedtoandcan<br />
initsturnserveasinputtootherneurons.Aconnectionbetweennodei<strong>of</strong> Agoodintroductioncanbefoundin(Haykin1994). Inthissectionwewillonlydescribethebasicingredients<strong>of</strong>neuralnetworks. 1.3<strong>Neural</strong><strong>Networks</strong><br />
wji.Thetotalinputujfornodejis acertainlayerandanodej<strong>of</strong>anotherlayerischaracterizedbyaweight<br />
wherexiistheactivation<strong>of</strong>aninputnodeiandthesummationrunsover allnnodesthatnodejreceivesinputfrom. uj=nXi=1wjixi (4)
8 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />
.<br />
.<br />
.<br />
.<br />
.<br />
.<br />
.<br />
.<br />
.<br />
.<br />
activation function<br />
summing<br />
function<br />
input<br />
signals<br />
synaptic<br />
weights<br />
treshold<br />
output<br />
θ<br />
φ(.)<br />
x<br />
x<br />
x<br />
w<br />
w<br />
w<br />
1<br />
2<br />
n<br />
j<br />
j<br />
1<br />
2<br />
n<br />
y<br />
u<br />
j<br />
j<br />
j<br />
j<br />
Figure7:Adiagram<strong>of</strong>neuronj.<br />
Nowthisnodeusesanactivationfunctiontodetermineitsownactivation<br />
andoutput yj=(ujj) (5)<br />
Hereisacertainactivationfunction,whichisusuallyasigmoidfunction<br />
forthehiddennodesandalinearfunctionfortheoutputnodeinregressionproblems,andjisacertainthresholdfornodej.Figure7showsa<br />
visualization<strong>of</strong>thisprocess.<br />
Thisprocedure<strong>of</strong>determiningtheinputandtheoutput<strong>of</strong>anodeisdonefor<br />
allnodes,excepttheinputnodes.Theinputnodesreceivetheiractivation<br />
directlyfromoutside(fromaleforexample).Inthiscasetheinputissome<br />
kind<strong>of</strong>representation<strong>of</strong>theformationmodel.<br />
Thetimethatisneededtodeterminetheoutput<strong>of</strong>theneuralnetfora<br />
certaininputcanbeexpressedinthenumber<strong>of</strong>connections(weights)<strong>of</strong>the<br />
net.Theoperationsdonebythenetareweightmultiplications,summations<br />
andcalculatingtheactivation.<br />
Determiningtheoutput<strong>of</strong>thenetwillnottakemuchtime.However,the<br />
process<strong>of</strong>makingthenetlearntheproblem,training,takesconsiderably<br />
moretime.When<strong>using</strong>neuralnetworks,thefollowingaspectsareimportant<br />
forthetrainingtimeandthegeneralizationability.Anetissaidtogeneralize<br />
wellifitproduces(nearly)correctoutputforinputthatwasnotusedduring<br />
itstraining.<br />
Thearchitecture<strong>of</strong>thenet.Largenets(highnumber<strong>of</strong>weights)learn<br />
slowlyandusuallydonotgeneralizewell.Thisisduetothefactthat<br />
theneuralnet\remembers"itstrainingexamplesifithastoomuch<br />
freedom(toomanyweights).
1.3<strong>Neural</strong><strong>Networks</strong> 9<br />
Thetrainingmethod.Severaltrainingmethodsforsupervisedlearningexist,forexampleback-propagation,conjugategradient,steepesingmeansthatthenetproducessomeoutput(theactualresponse)<br />
certainparametersisveryimportantanddicult.Supervisedlearn-<br />
andcorrectsitsbehaviouraccordingtothecorrectoutput(desiredre-<br />
descent,momentumdescent.Forsome<strong>of</strong>thesemethodsthetuning<strong>of</strong> sponse).Formultilayerfeedforwardnetworks(networksthathavecon-<br />
nectionsdirectedfromtheinputtotheoutput),themostwidelyused algorithmistheback-propagationalgorithm.Therearetwophases intheBP-learning.Intherstphase,theforwardphase,theinputsignalspropagatethroughthenetworklayerbylayer,eventually<br />
responsesoproducediscomparedtothedesiredresponse.Theerror signalsgeneratedbythiscomparisonarethenpropagatedbackwards producingsomeresponseattheoutput<strong>of</strong>thenetwork.Theactual<br />
Thetrainingset.Thetrainingsetshouldberepresentativeforthe throughthenetinthesecond,thebackwardphase.Moreinformation<br />
problem,otherwisethenetisnotabletolearntheproblemortogeneralizewell.Withconjugategradienteachtrainingexampleisevaluated<br />
ontrainingmethodscanbefoundin(Haykin1994).<br />
duringtraining(batchtraining),soalargetrainingsetcausesalong<br />
1.3.1Networkarchitecture inSection1.3.3. trainingtime.Moreonthecreation<strong>of</strong>thetrainingsetcanbefound<br />
perlayerandtheconnectionsbetweenthelayers;itdescribeswhatthenet Thenetworkarchitectureisadescription<strong>of</strong>thelayers,thenumber<strong>of</strong>nodes nodes<strong>of</strong>onelayerreceiveinputfromallnodesinthepreviouslayer.This lookslike.Forexampleafullyconnectedneuralnetisanetinwhichthe<br />
cationsforthelearningandgeneralizationability<strong>of</strong>thenet. Anothermethodisalocallyconnectednetwork,whichisdescribedlater Thearchitecturedeterminesthenumber<strong>of</strong>weightsandhascertainimpli-<br />
isthemostcommonarchitecture,butitleadstoalargenumber<strong>of</strong>weights. whenweturntoreceptiveeldsandconvolutionalnetworks.<br />
1.3.2TheLearningMethod Anetusesalearningmethodtominimalizetheerrorinthenet.Thetraining setconsists<strong>of</strong>Ppatterns(xp;dp).Here,xpistheinputpatternpanddp isthedesiredoutputforthisinputpattern.Foracertaininputxpthe networkcalculatesanoutputap.Theerrorisgivenbythequadraticsum
10 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />
<strong>of</strong>thedierence<strong>of</strong>thisoutputandthedesiredoutputdp:<br />
isthattheweightsareadaptedduringtrainingsothatEisminimalized. Here,Pisthenumber<strong>of</strong>examples.Theideabehindthelearningmethods E=PXp=1(dpap)2 (6)<br />
adaptedinthedirectionwiththesteepestdescent Thisisdonebyadownhilltechnique,gradientdescent.Theweightsare 0B@Weight<br />
wji1CA=0B@learning-rate parameter 1CA0B@local Theweightcorrectiondependsonalearningparameter,thelocalgradient gradient1CA0B@inputsignal <strong>of</strong>neuronj toneuroni 1 CA(7) andtheinputsignal<strong>of</strong>neuronitoneuronj.IfEnolongerdecreasesthe<br />
betweenthedirectionandthegradientvectors.TheCGmethodisthemost bythegradientdescentmethod,byincorporatinganintricaterelationship <strong>of</strong>thenormalgradientdescentmethod.Itavoidsthezigzagpathfollowed trainingstops.<br />
convenientalgorithm,becauseitneedsnotuningparametersanditisfaster Wehaveusedtheconjugategradient(CG)method,whichisanadaptation<br />
thannormalback-propagationifthesize<strong>of</strong>thetrainingsetisnottoolarge. WehaveusedtheXerion(version3.1and4.0)simulatortotrainthenetworks (formoredetailsonneuralnetworksimulatorsseeAppendixA).<br />
layer.Howthisisdone,isdiscussedinthenextchapter.Nowwewillfocus Theinputhastobepresentedtothenet,sothephysicalmodelmustbe transformedintoaset<strong>of</strong>numbersthatfunctionasactivations<strong>of</strong>theinput 1.3.3Thetrainingset<br />
onthescaling.<br />
theweightswillingeneralbefarapartaswell.Thiswillslowdownthe alsohavealargevariation.Soiftheinputsdonotlieinasmallrange, inputshaveaveryhighratio(forexamplei1=1andi2=1000)theweights range[1;1].Aswehaveseenthenetcalculatesaweightedinput.Iftwo Theinputtothenetmustbescaledsothevaluesliemoreorlessinthe<br />
set<strong>of</strong>weights.Anotherpoint<strong>of</strong>viewisthatlargeinputvalueswillhave training,becausealargerweightspacehastobesearchedforanoptimum moreinuenceontheactivation<strong>of</strong>thenodestheyareconnectedto.Inthis waywehavealreadybuildsomepriorknowledgeintothenet.Toavoidthis, wescalealltheinputstoasmallrange.Whencertaininputsareimportant, inSection2.1.3. thenetcanlearnthatitself.Moreonthissubject<strong>of</strong>scalingcanbefound
1.3<strong>Neural</strong><strong>Networks</strong> 11<br />
Atrainingsetshouldbesucientlylarge.Althoughthereisnogeneral prescription<strong>of</strong>howlargeatrainingsetshouldbe,therearesome\rules", likethefollowingfromBaumandHassler(Haykin1994).Anetworkwill followingtwoconditionsaremet: almostcertainlyprovidegeneralization(seenextsection),providedthatthe<br />
2.Thenumber<strong>of</strong>examples,Pusedintrainingis 1.Thefraction<strong>of</strong>errorsmadeonthetrainingsetislessthat2.<br />
Here,Wisthetotalnumber<strong>of</strong>weightsandMisthetotalnumber P32Wln32M<br />
<strong>of</strong>hiddennodes.Thisformulaprovidesadistribution-free,worstcaseestimationforthesize<strong>of</strong>thetrainingsetforasingle-layerneural<br />
(8)<br />
Atrainingsetshouldalsoberepresentative.Thismeansthattheexamples inthetrainingsetarerandomlygeneratedanddistributedoverthewhole inputspace. networkthatissucientforagoodgeneralization.<br />
1.3.4Generalization<br />
bythreefactors: isdeterminedbyitsabilitytopredictoutputstoinputsithasnotseen duringtraining,whichiscalledgeneralization.Generalizationisinuenced Althoughaneuralnetcanlearnanyinput-outputmapping,itsapplicability<br />
thesizeandrepresentativeness<strong>of</strong>thetrainingset,<br />
Withto<strong>of</strong>ewexamplesthenetjustmemorizesthetrainingsetandexhibits thephysicalcomplexity<strong>of</strong>theproblemathand. thearchitecture<strong>of</strong>thenetwork,<br />
poorgeneralization.Ifthenumber<strong>of</strong>examplesismorethanthenumber <strong>of</strong>weights,thenetwillgeneralizebetter.Widrow'srule<strong>of</strong>thumb(Haykin setsize<strong>of</strong>approximately10timesthenumber<strong>of</strong>weightswhentheerroron thetrainingsetis10%. Inourproject,wewillcomparedierentarchitecturesononespecictrainingset.Thearchitecturethatgivesthebestresults,interms<strong>of</strong>training<br />
error,generalization(testing)errorandcomplexity(number<strong>of</strong>weights),will<br />
1994)comesfromequation8andstatesthatinpracticeweneedatraining<br />
betrainedonalargertrainingset.
12 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />
global minimum<br />
Error surface<br />
local minimum<br />
local minimum<br />
InFigure8(left)theerrorsurface,belongingtospecicweightvectors, 1.3.5Localminima Figure8:Localminimaintheerrorsurface.<br />
global minimum<br />
local minimum<br />
initialization<br />
technique,whichmaycausethealgorithmtogettrappedinalocalminimum isshown.Thelearningalgorithmthatweuseisbasicallyahill-descending<br />
minimum<br />
errorsmallerthanthepreviousone.Butsomewhereelseintheweightspace thereexistsanotherset<strong>of</strong>synapticweightsforwhichthecostfunctionis intheerrorsurface,althoughweareinterestedintheglobalminimum.The algorithmgetstrapped,becauseitcannotndadirectionwhichmakesthe<br />
tializations.ThisprocessisshowninFigure8(right);withdierentweight smallerthanthelocalminimuminwhichthenetworkisstuck.Onemethod toavoidlocalminimaisretrainingtheneuralnetwithdierentweightini-<br />
initializationthenetwillconvergetodierentminima.<br />
andtheShallow-log.Theformationisdescribedbyanumber<strong>of</strong>beds.For eachbedanumber<strong>of</strong>radialzonesaregivenandforeachradialzonethe 1.4<strong>Forward</strong><strong>Modelling</strong><strong>using</strong>a<strong>Neural</strong>Network Thegoal<strong>of</strong>thisprojectistoobtainagoodapproximation<strong>of</strong>theDeep-<br />
resistivity(m)anditssize(inch)isgiven.Therstradialzoneisthebore<br />
Thetoolresponseconsists<strong>of</strong>twocontinuouslogs(theShallow-logandthe thetrueresistivity.Allthiscanbedescribedwithamodellikeshownin holewithitsradiusandtheresistivity<strong>of</strong>thedrillinguid,thenextradial<br />
Deep-log)likeshowninFigure10andTable1.ThemodelshowninFigure Figure9(thecorrespondingformationisshownontherightside). zonedescribestheinvasion(ifthereisany)andthelastradialzonedescribes<br />
innity(thisisthesameforallmodels,indicatinganinniteshoulderbed). 9contains80beds.Therstbedhas3radialzonesanditstartsatminus<br />
aresistivity(Rt)<strong>of</strong>27m.Thelastradialzonehasaradius<strong>of</strong>innity 45inchandaresistivity(Rxo)<strong>of</strong>1.90m.Andnallythethirdzonehas andthemudresistivityis0.05m.Thesecondzonehasaradius(dxo)<strong>of</strong> Therstradialzone<strong>of</strong>thisbed(theborehole)hasaradius<strong>of</strong>4.25inch
1.4<strong>Forward</strong><strong>Modelling</strong><strong>using</strong>a<strong>Neural</strong>Network 13<br />
number <strong>of</strong> beds<br />
borehole radius<br />
4.25 inch<br />
(resistivity drilling fluid<br />
0.05 Ωm)<br />
1e+09<br />
Rt = 27 Ωm<br />
80<br />
3<br />
-1e+09<br />
one bed<br />
dxo = 45 inch<br />
0.05<br />
4.25<br />
1.90<br />
27.00<br />
45.00<br />
1e+09<br />
Rxo = 1.90 Ω m<br />
start depth <strong>of</strong><br />
bed<br />
number <strong>of</strong><br />
3 0.00<br />
Rt = 12 Ωm<br />
radial<br />
borehole<br />
zones<br />
0.05 4.25 radius<br />
0.90 19.00<br />
dxo = 19 inch<br />
invasion<br />
resistivity<br />
radius<br />
12.00 1e+09<br />
<strong>of</strong> drilling<br />
fluid<br />
(otherwisetherewouldbeanotherradialzone).<br />
radius <strong>of</strong><br />
Theneuralnetcanbeusedasafastforwardmodelwhentheaveragerelative Figure9:Aformationanditsmodel.<br />
virgin errorbetweentheforwardmodeloutputdandtheneuralnetworkoutputa<br />
invasion<br />
true<br />
Rxo = 0.90 Ω m<br />
liesbelow5%.<br />
resistivity<br />
resistivity<br />
depth 0.0.110610E+02.330318E+01 0.2.108760E+02.325301E+01 0.4.106962E+02.320733E+01 Table1:Thetoolresponse.<br />
0.6.105092E+02.316235E+01 LLd LLs<br />
radiusandresistivity<strong>of</strong>thedrillinguidarexedto4.25inchand0.05m Theearthmodelsinthisprojectarecreatedbyassigningrandomnumbers withinacertainrangefortheparametersRt,Rxoanddxo.Theborehole . . .<br />
forwardmodelthatisusedatKSEPL. respectively.Thetoolresponsestothesemodelsarecalculatedwiththe
14 1.FORWARDMODELLINGOFRESISTIVITYLOGS<br />
Ohm meter<br />
True resistivity Rt<br />
Deep-log<br />
Shallow-log<br />
(LLd)andShallow-log(LLs).TheinputsignalsRxoanddxoareomitted. Figure10:Part<strong>of</strong>amodelwithinputsignalRtandoutputsignalsDeep-log<br />
100<br />
10<br />
5400 5420 5440 5460 5480 5500<br />
depth (feet)
15<br />
point <strong>of</strong> interest<br />
(window is centered around this point)<br />
Tool response<br />
True resistivity Rt<br />
Invasion resistivity Rxo<br />
AsdescribedinSection1.3,thetrainingtimemainlydependsonthesize<strong>of</strong> increasethegeneralizationcapability,weliketokeepthenetassmallas thenetandthesize<strong>of</strong>thetrainingset.Todecreasethetrainingtimeand 2Inputrepresentationandarchitecturedesign<br />
possible.Thiscanbeachievedbychoosingacompactinputrepresentation andbydecreasingthenumber<strong>of</strong>freevariables(weights)inthenet.The inputrepresentationhastobecompactandappropriatetotheproblem. T<strong>of</strong>acilitatethelearningprocess,weemploytwomethods.Firstlywepreprocesstheinputandsecondlyweforcecertainconstraintsonthenetwork<br />
methodcontributest<strong>of</strong>acilitatinglearningonlyiftheresultingstructure networkwhilemakingonlyasmallcomputationaloverhead.Thesecond reectsthedesigner'saprioriknowledge<strong>of</strong>theproblem.Otherwisethe aterepresentation<strong>of</strong>theinputwhichsimpliestheproblemfortheneural architecture.Thepurpose<strong>of</strong>therstmethodistocreateanintermedi-<br />
2.1Inputrepresentation networkisaprioribiasedtowardswrongsolutions.<br />
Wearelookingforacompactandappropriateinputrepresentation.Given anearthmodel,describedbyn(2+r2)+1values(fornbeds<strong>of</strong>rradial<br />
Figure11:Example<strong>of</strong>slidingwindowinputrepresentation.Aslidingwindow<strong>of</strong>sizewisplacedaroundthepoint<strong>of</strong>interestalongtheinputmodels.<br />
Invasion diameter dxo<br />
w
16 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />
(thetargetisalsocalledthedesiredoutput).Foroneformationwehavea toolresponse<strong>of</strong>mfeet,sampledeverytfeet.Thisproducesmt+1targets Thetoolresponseataspecicdepthisusedasthetarget<strong>of</strong>theneuralnet perlog.Butwhatdoweuseasinputtoproducethistarget?Weassume zones),thenetshouldbeabletoproducethetoolresponseatanydepth.<br />
slidingwindow.TheslidingwindowapproachisshowninFigure11. forthetoolresponseatthatdepth.Thispart<strong>of</strong>theformationiscalleda thatapart<strong>of</strong>theformation,centeredaroundacertaindepth,isresponsible<br />
Therstmethodisquitestraightforward.Itsamplesthepart<strong>of</strong>theformationthatliesinthewindow,without<strong>using</strong>knowledgeabouttheinput<br />
orrelationsbetweeninputs. Thesecondmethodlooksmoreliketheoriginalmodeldescriptionanduses thefactthattheformationisdescribedbybeds.Inthemodeleachbedis centeredaroundthepoint<strong>of</strong>interestbyanumber<strong>of</strong>features. describedbyanumber<strong>of</strong>values,whichcouldbeseenasattributes<strong>of</strong>that<br />
modelintheslidingwindowandbydescribingthebedsthatlieinthe Theslidingwindowisdescribedintwoways:bydiscretizingtheformation<br />
2.1.1Discretizedslidingwindow bed.Intheattributesapproachwedescribethebedsthatlieinawindow,<br />
Whenthereisinvasion,seeSection1.1.2,theformationmodelcontainsthree<br />
inputswhenthereisnoinvasionand3timesthisnumberwhenthereis sampledwithinthiswindowwithasamplingperiods,resultinginws+1 isplacedalongtheinputlogsaroundthepoint<strong>of</strong>interest.Themodelsare containsonlythevariableRt.Weuseaslidingwindow<strong>of</strong>xedsizew,which variablesRt,Rxoanddxo.Whenthereisnoinvasiontheformationmodel<br />
<strong>of</strong>theslidingwindow: invasion. Thefollowingaspectsshouldbetakenintoaccountindeterminingthesize Thesize<strong>of</strong>thetool.Thecurrentsowingfromthetoolpenetrate theformation.Thecurrents<strong>of</strong>theShallowLaterologpenetratethe formationandreturntothetopandbottom<strong>of</strong>thetool.Thepart<strong>of</strong><br />
Thetype<strong>of</strong>targetlog(DeeporShallow).AsshowninFigure3,the largeasthetoolitself.Thetoolisapproximately30ft,whichgives theformationthatthetoolreceivesinformationfrom,isatleastas<br />
DeepLaterologreceivesinformationfromalargerpart<strong>of</strong>theformation.Thecurrents<strong>of</strong>theShallowLaterologreturntothetoolitself<br />
anindicationthatthewindowsizeshouldalsobeatleast30ft.<br />
windowsizefortheDeep-logshouldbelargerthanfortheShallow-log anddonotpenetratetheformationmuch.Thisindicatesthatthe
Ω meter<br />
60<br />
45<br />
30<br />
input model Rt<br />
15<br />
2.1Inputrepresentation 17<br />
Thesize<strong>of</strong>thebeds.Whathappensifthewindowissmallerthan thelargestbedintheformationisshowninFigure12.Thesliding windowislocatedmorethanonceinthesamebed,producingthe sameinput,butpossiblynotthesametarget.Nowtheneuralnet hastolearnf(x)=y1andf(x)=y2,makingthetheproblemnondeterministic.Conictingexamplesmakeitverydicultforthenet<br />
dierenttargetsforoneinput. tolearntheproblem,becauseitadaptsitsweightstoreproducetwo<br />
0<br />
second (first<br />
model,theinputtothenetwillbethesameforthesketchedsituation, Figure12:Whentheslidingwindowissmallerthanthelargestbedinthe<br />
shifted by some samples)<br />
to net looks like<br />
30 30 30 30 .... 30 30 30 30<br />
WeexpectthenetthatistrainedontheShallow-logwillperformbetter, becauseitneedsasmallerwindowthantheDeep-log.Thesamplingperiod althoughthetargetdiersalot.<br />
first window, to looks like<br />
30 30 30 30 ... 30 30 30 30<br />
<strong>of</strong>thetargetlogdeterminesthenumber<strong>of</strong>examplesweobtainperlog.In reasons,wehavetakenveloggingpointsperfeet.Thesamplingperiodin theslidingwindowisimportantfortheresolution<strong>of</strong>thebedboundaries. realapplicationsonetakestwologgingpointsperfeet,butforeciency<br />
windowasforthetargetlog.Inthiscasethatwouldmean<strong>using</strong>asampling withsfeetresolution.Whenwewanttohaveatleastthesameaccuracy period<strong>of</strong>0.2feet. Whenweuseasamplingperiod<strong>of</strong>sfeet,wecandescribeabedboundary<br />
Thisinputrepresentationisnotverycompact.Whenweuseaslidingwindow<strong>of</strong>25.4feetandasamplingperiod<strong>of</strong>0.2feet,wehave384inputs(128<br />
asthetargetlog,weshouldusethesamesamplingperiodforthesliding<br />
pervariableand3variableswhenthereisinvasion).
18 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />
bexedforeachsample.Weuseaxedsizewindowanddescribethebeds 2.1.2Attributes Theinputmodeldescribesanumber<strong>of</strong>bedboundaries,eachdescribedby anumber<strong>of</strong>radialzones.Theneuralnetrequiresthenumber<strong>of</strong>inputsto thatoccurwithinthiswindow.Ifthewindowcontainslessthanthexed shoulderbeds.Theorderinwhichthebedsarepresentedtotheneuralnet number<strong>of</strong>beds,weadd\default"beds.Thesebedsfunctionasinnite<br />
thisbedarepresentedandthebedsadjacenttothosebedsandsoon. shouldbexed(inourcaseitispresentedrst).Thenthebedsadjacentto Eachbedisdescribedbyanumber<strong>of</strong>attributes(alsocalledfeatures).The thetargetsignalforexample(probablythebedinthecenter<strong>of</strong>thewindow) isimportant.Thelocationintheinput<strong>of</strong>thebedthathasmostinuenceon<br />
contrastbetweentwobedsisdenedasv1=v2andthedierenceasv1v2 forvaluesv1andv2.Theattributesweuseare: 1.thetrueresistivity<strong>of</strong>thebed(Rt);<br />
4.theinversedistancetotheloggingpoint(pointsclosetothebedboundaryareconsideredtobemoreimportantthanpointsthatliefurther<br />
away); 3.theinvasionradius<strong>of</strong>thebed(dxo); 2.theinvasionresistivity<strong>of</strong>thebed(Rxo);<br />
5.thecontrastbetweenthetrueresistivity<strong>of</strong>thisbedandthebedthat 6.thecontrastbetweentheinvasionresistivity<strong>of</strong>thisbedandthebed<br />
7.thecontrastbetweentheinvasionradius<strong>of</strong>thisbedandthebedthat thatliesbelowthisbed;<br />
8.thedierencebetweenthetrueresistivity<strong>of</strong>thisbedandthebedthat<br />
9.thedierencebetweentheinvasionresistivity<strong>of</strong>thisbedandthebed<br />
10.thedierencebetweentheinvasionradius<strong>of</strong>thisbedandthebedthat thatliesbelowthisbed;<br />
centbeds.Theinversedistancetotheloggingpointis0forthedefaultbeds Thedefaultbedshavenocontrast(1)andnodierence(0)withtheiradja-<br />
(thebedscontinuetoinnity).Ifwedescribenbedsinthechosenwindow, liesbelowthisbed.<br />
thisresultsin10ninputs.
2.1Inputrepresentation 19<br />
Thisapproachismorecompactthanthediscretizedslidingwindowapproach,butitisdiculttochooseappropriatefeaturesthatwillfacilitate<br />
problem,whenthesefeaturesarenotdescribingtheproblemwell. befacilitated.Itcouldevenmakeitdicultfortheneuralnettolearnthe thelearningprocess.Weassumethecontrastsanddierencesareimportant anddescribetheproblemwell.Ifthisisnotthecase,thelearningwillnot<br />
combination<strong>of</strong>logarithmicandnormalizationscalingandfortheothervariablesanormalizationscaling.Thetrueresistivity(andthetoolresponse)<br />
Asdiscussedintherstchapter,theinputshouldbeapproximatelyscaledto thedomain[1;+1].Forthetrueresistivityandthetoolresponseweusea 2.1.3Inputandoutputscaling<br />
canrangebetween1and2000m.Toreducethisrange,weusealogarithmicscaling.Therange<strong>of</strong>theothervariables,theinvasionresistivity<br />
logarithmicscalingonthesevariables.Thescalingstakethefollowingform andtheinvasionradius,aremuchsmallerandthereforewedonotusethe<br />
where,xistheoriginalinputoroutputvalue,themeanandthestandard x0norm=x x0log=ln(x) (10) (9)<br />
deviation<strong>of</strong>thevariablex.Thefactorisappliedtomakethescaledrange evensmaller(intheexperiments=2). AnormalizedGaussiandistributionisshowninFigure13.Fromthisgure wendthatfor=1,68.27%<strong>of</strong>thevalues(<strong>of</strong>thisspecicdistribution) liebetween-1and+1.Whenweuse=2wendthat95.45%<strong>of</strong>the valuesliebetween-1and+1.<br />
-3 -2 -1 1 2 3<br />
applied. Intheattributedescriptionapproachonlyanormalizedscalingmethodis Figure13:Standardnormaldistribution.<br />
68.27 %<br />
95.45 %<br />
99.73 %
20 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />
quencefortheminimalization<strong>of</strong>thenetworkerror.Formoredetailsonthe Thiscombinedscalingmethod<strong>of</strong>thetoolresponsehasanattractiveconse-<br />
2.2Preprocessing minimalization<strong>of</strong>thenetworkerrorseeSection2.4.<br />
Inthissectionwewilldescribethepreprocessingmethodswehaveusedin theproject.Thepurpose<strong>of</strong>preprocessingistondanintermediateinput representationthatfacilitatesthelearningprocess.Anotheradvantage<strong>of</strong><br />
Rt,Rxoanddxoandsecondlyasthreefunctions<strong>of</strong>thedepthx,Rt(x), windowapproach. Wecanviewtheinputintwoways.FirstlyasthreeN-dimensionalvectors wasonlyappliedtotheinputthatwascreatedbythediscretizedsliding preprocessingisthatwecanreducethenumber<strong>of</strong>inputs.Thepreprocessing<br />
Rxo(x)anddxo(x).Thelatterisactuallyalsoavector,becausethefunctionsareequallysampledwithinaninterval[1;N].Intherstcasewe<br />
projecttheinputtoanM-dimensionalsubspacespannedbytheprinciple components<strong>of</strong>theinput.Intheothercaseweuseaset<strong>of</strong>orthogonalbasis functions,theHaarwavelets,toprojecttheinput. Thedisadvantage<strong>of</strong>inputreductionistheloss<strong>of</strong>information.Hopefully, theinformationthatislosthasanegligibleinuenceonthelearningand generalization<strong>of</strong>thenet. 2.2.1PrincipalComponentAnalysis Thefollowingdescription<strong>of</strong>theprincipalcomponentanalysisistakenfrom (Hertz,Krogh&Palmer1991).<br />
ProjectingthedatafromtheiroriginalN-dimensionalspaceontotheMdimensionalsubspacespannedbythesevectorsperformsadimensionality<br />
reductionthatretainsmost<strong>of</strong>theintrinsicinformationinthedata. inthesubspaceperpendiculartotherst.Withinthatsubspaceitistaken maximumvariance.Thesecondprincipalcomponentisconstrainedtolie alongthedirectionwiththemaximumvariance.Thenthethirdprincipal Therstprincipalcomponentistakentobealongthedirectionwiththe componentistakeninthemaximumvariancedirectioninthesubspace alonganeigenvectordirectionbelongingtothekthlargesteigenvalue<strong>of</strong>the Ingeneralitcanbeshownthatthekthprincipalcomponentdirectionis<br />
analysis(PCA),alsoknownastheKarhunen-Loevetransformincommunicationtheory.Theaimistondaset<strong>of</strong>Morthogonalbasisvectors<br />
Acommonmethodfromstatisticsforanalyzingdataisprincipalcomponent (eigenvectors)thataccountforasmuchaspossible<strong>of</strong>thedata'svariance.<br />
perpendiculartothersttwo,andsoon.AnexampleisshowninFigure14.
2.2Preprocessing 21<br />
ε<br />
1<br />
e1<br />
ε<br />
2<br />
e2<br />
covariancematrix.ThismatrixiscalculatedforPpatternsby Figure14:Most<strong>of</strong>thevariation<strong>of</strong>thedataliesinthedirection<strong>of</strong>1.<br />
Covariance(i;j)=Pp(xpixi)(xpjxj)<br />
e3<br />
Here,xpiisinputi<strong>of</strong>patternp,xpjisinputj<strong>of</strong>patternpandxiand xjarethemeans<strong>of</strong>inputiandinputjrespectively.Thenthematrixis diagonalizedandtheeigenvaluesarecalculated. Theoriginalinputvectorx(Rt;Rxoordxo)iswrittenas wheree1,...,eNaretheoriginalbasisvectorsasshowninFigure14.We canalsowritetheinputvectorinanotherorthogonalset<strong>of</strong>basisvectors,the eigenvectors1;:::;Nx=x1e1+x2e2+:::+xNeN (12)<br />
Toreducethesize<strong>of</strong>thevectorsfromNtoM,weprojectthisvectorto x=(1x)1+(2x)2+:::+(Mx)M+:::+(Nx)N x0=(1x)1+(2x)2+:::+(Mx)M (13)<br />
theneuralnet. Instead<strong>of</strong><strong>using</strong>Nvaluesx1;x2;:::;xNfromtheoriginalinputvectorweuse Mvalues1x;2x;:::;Mx<strong>of</strong>theprojectedinputvectorasinputsfor (14)<br />
P (11)
22 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />
Tocalculatethepercentage<strong>of</strong>informationthatislostbythisprojection,we rsthavetowritex0intheeibasisvectors: Thevaluesaiarecalculatedby x0=a1e1+a2e2+:::+aNeN (15)<br />
patternsintheinputle)by Theinformationthatislostbythisprojectioncanbecalculated(forP ai=NXn=1xnMXm=1mnmi (16)<br />
Loss<strong>of</strong>information=1PPXp=1jjx0pxpjj<br />
2.2.2Wavelettransform jjxpjj (17)<br />
onalbasisfunctionsfi(x) Wecanwriteanyfunctionf(x)asaweightedcombination<strong>of</strong>otherorthog-<br />
f(x)isacontinuousfunction,weusethecoecientsci.Wechoosefunctions Instead<strong>of</strong><strong>using</strong>thevalues<strong>of</strong>f(x)(asinputs),whichmaybeinnitewhen f(x)=Xicifi(x) (18)<br />
fi(x)withproperties,thatmakeiteasiertomanipulatewiththosefunctions AcommonlyknownmethodistheFouriertransform,wheretheorthogonal thanwiththeoriginalfunctionf(x). basisfunctionsaresin(ax)andcos(ax).Thesebasisfunctionsallowyouto describethefunctionondierentfrequencylevels.<br />
anotherinterestingset<strong>of</strong>orthogonalbasisfunctions,calledwavelets.Weare OurinputsignalsRt(x),Rxo(x)anddxo(x)haveaveryspecialshape:they<br />
especiallyinterestedinthesimplestwavelets,theso-calledHaarwavelets. inthiscase,becausewewouldneedaninnitenumber<strong>of</strong>coecientsand basisfunctionstocorrectlymodelthediscretetransitions.Thereishowever areallrectangular\functions".TheFouriertransformisnotappropriate<br />
part<strong>of</strong>thewindow(varyinginsizeandlocation).Thepropertytheydescribeisthedierencebetweentheaveragevalueovertherstandseconestingaboutwaveletsisthattheinputcanbedescribedondierentdetail<br />
levels.Allcoecients,excepttherst,describeaspecicpropertyovera ThesewaveletsareblockfunctionsasshowninFigure16.Whatissointer-<br />
half<strong>of</strong>theirpart<strong>of</strong>thewindow.Therstcoecientdescribestheaverage overthewholewindow.
2.2Preprocessing 23<br />
Wewillrstgiveashortintroductiononwaveletsandhowthecoecients AwaveletisdenedbyW(x)=Xk(1)kc1k(2xk)<br />
arecalculated(takenfrom(Strang1989))andthenwewillexplainwhythe waveletsaresousefulinourproject.<br />
by Here,kistakensymmetricallyaroundzero.Thescalingfunctionisdened (19)<br />
underconditions (x)=Xck(2xk) Zdx=1 (20)<br />
TheHaarwaveletisthesimplestwavelet.Forthiswaveletwechoosec0=1 Xkck=2 (21)<br />
andc1=1.Thescalingfunctionforthesecoecientsisablockfunction denedby (22)<br />
ThewaveletasshowninFigure15,isdescribedby (x)=(10x1<br />
WHaar(x)=(2x)(2x1) 0otherwise (23)<br />
Westartwithavector(f1;f2;:::;fN),whichareN=2jequallysampled (24)<br />
Figure15:HaarwaveletfromboxfunctionWHaar(x)=(2x)(2x1). values<strong>of</strong>afunctionf(x)onaunitinterval.Thisvectorwillbeapproximatedbythesum<strong>of</strong>dierentweightedblockfunctions.InFigure16the<br />
0 1<br />
rst8blockfunctions,belongingtothelevels0,1and2,aregiven.The projectedinputvectorisdescribedbythecoecientsthatcorrespondtothe
24 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />
weighing<strong>of</strong>theseblockfunctions. Howarethesecoecientscalculated?Onavectorx<strong>of</strong>2jvaluesweperform L(x),calculatesthemeanandthesecond,H(x),calculatesthedierence. twooperationsL:R2j!R2j1andH:R2j!R2j1.Therstoperation,<br />
L(x)=k0 B @x1+x2 xN1+xN x3+x4 . 1 CA H(x)=k0 B @x1x2 x3x4<br />
Here,usuallyk=12indecompositionandk=1inreconstruction,but xN1xN . 1 CA (25)<br />
wecouldalsousek=12p2inboththedecompositionandreconstruction, doneintheexperimentsdescribedlater). ThevectorsproducedbyL(x)andH(x)arebothhalfthesize<strong>of</strong>theoriginal whichhastheadvantage<strong>of</strong>normalizingthewaveletsateveryscale(thisis<br />
level,levelj1.Tondthecoecientsatthenextdetaillevel,weperform vector.ThecoecientsfoundbyH(x)arethecoecientsonthenestdetail atthepreviouslevel.Thiscontinuesuntilwereachlevel0. H(x)andthecoecientatlevel0foundbyL(x).So,ifwenameLithe averageoperatoratleveliandHithedierenceoperatoratleveli,thenew theoperationsL(x)andH(x)recursivelyonthevectorproducedbyL(x) Theprojectedinputconsists<strong>of</strong>thecoecientsperdetaillevelfoundby Onlevelithereare2icoecientsproducedbyHi,thedierenceoperator. inputvectorisconstructedby(L0;H0;H1;:::;Hj1). Onlevel0wehaveanextracoecientcomingfromL0.Thetotalnumber <strong>of</strong>inputsis whichisequaltothesize<strong>of</strong>theoriginalinputvector. Wewillnowpresentanexampleforavector<strong>of</strong>length23,x=(1;3;2;2;5;3;8;0). N=1+20+21+:::+2j1=2j (26)<br />
Intherststeponlevelj=2,wegetfork=12 L(x)=0B@24 1 CAandH(x)=0B@1<br />
Wecontinueonlevelj=1withL(x)=x0asinputvector 014 1 CA (27)<br />
L(x0)= 24!andH(x0)= 0! (28)
2.2Preprocessing 25<br />
=<br />
C<br />
C<br />
0<br />
1<br />
x<br />
x<br />
+<br />
+<br />
C<br />
2<br />
x<br />
+<br />
C<br />
3<br />
x<br />
+<br />
C<br />
4<br />
x<br />
+<br />
C x<br />
+<br />
5<br />
C x<br />
+<br />
net.<br />
6<br />
Andnallyonlevelj=0wendforL(x0)=x00 Figure16:Theinputsignaliswrittenasaweightedcombination<strong>of</strong>rectangularblockfunctions.Thecoecientsciareusedasinputstotheneural<br />
C<br />
7<br />
L(x00)=3andH(x00)=1<br />
x<br />
Thevector<strong>of</strong>waveletcoecients,f0(x)is(L(x00);H(x00);H(x0);H(x)) (29)<br />
f(x)= 0 B@ 1 325380 1 CAandf0(x)= 0 B@ 1 3 014 1 CA (30)
26 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />
thewindowismoreimportantthantheinformationontheedges.Weknow exactlywhichcoecientsarecorrespondingtotheedgesandwhichtothe Whatissointerestingaboutthewaveletcoecients,isthattheydescribea center<strong>of</strong>thewindowandwecanremove(some<strong>of</strong>)thecoecientsthatare verylocalarea<strong>of</strong>theinput.Weexpectthattheinformationinthecenter<strong>of</strong><br />
coecient,thetransitionsintherstquarter<strong>of</strong>thewindowcanonlybe describedbysteps<strong>of</strong>w=4<strong>of</strong>coecientc2.Inthiscasewelooseresolution Coecientc4fromFigure16,forexample,describesthetransitionsinthe correspondingtotheedges,whenwearenotinterestedinthisinformation.<br />
Ifwehaveaninputcomingfromaslidingwindow<strong>of</strong>w=25.4feet,sampled ontherstquarter<strong>of</strong>thewindow. rst14<strong>of</strong>thewindow(sizew)withsteps<strong>of</strong>sizew=8.Ifweremovethis<br />
every0.2feet,wehave128inputvaluesand7detaillevels0to6.Onthe nerdetaillevels(5and6)weremoveanumber<strong>of</strong>coecientsontheedges aredescribedbysteps<strong>of</strong>0.2feetandontheedgesbysteps<strong>of</strong>0.8feet(this isdetaillevel4). InFigure17isshownwhathappenswhenanumber<strong>of</strong>coecientsonthe accuracyastheoriginalinputinthecenter.Inthecenterthetransitions <strong>of</strong>thewindow.Welooseresolutionattheedges,butweretainthesame<br />
edgesisremoved.Thetransitionsinthecenteraredescribedveryaccurately, butontheedgesweuselargerstepsthanintheoriginalinput(atransition isthenapproximatedbystepsinstead<strong>of</strong>onetransition).<br />
center <strong>of</strong> sliding window<br />
accurately. Figure17:Boundariesoutsidethecenter<strong>of</strong>thewindowarenotdescribed<br />
approximation<br />
true boundaries
2.3Architectureconstraints 27<br />
tailedinformationinthecenterispreserved.If,forexample,weremove20 5,weremoved7.8feetonlevel6andalso7.8feetatlevel5.Thismeans thatattheseparts<strong>of</strong>thewindowwehavearesolution<strong>of</strong>0.8feetandinthe Thecoecientsareremovedfromtheedges<strong>of</strong>thewindow,soallthede-<br />
coecientsoneachsideonlevel6and10coecientsoneachsideonlevel center9.8feetwehavearesolution<strong>of</strong>0.2feet. Howmuchinformationislostbythisoperationdependsonthelocation <strong>of</strong>thebedboundariesandthesharpness<strong>of</strong>thetransitions.Aresistivity contrastisapproximatedbysmallstepsinstead<strong>of</strong>onetransition.Ahard duringtrainingandtesting. imationaectsthegeneralizationperformanceinanegativewaywillshow measurefortheloss<strong>of</strong>informationisdiculttogive.Whethertheapprox-<br />
2.3Architectureconstraints Thesecondmethodtoreducethecomplexity<strong>of</strong>thenet,aswehadalready onthearchitecture.Theadvantage<strong>of</strong>thismethodisthatnoinformation mentionedinthebeginning<strong>of</strong>thissection,isforcingcertainconstraints islost,likeinthepreprocessingmethods.Wecanalsoreectourprior knowledgeabouttheinputinthedesign<strong>of</strong>thenet.Thisfacilitatesthe By<strong>using</strong>locallyconnectedneurons(receptiveelds),thenetcontainsmuch learningprocessandhopefullywillimprovethegeneralization<strong>of</strong>thenet.<br />
architectures. advantagesanddisadvantages<strong>of</strong>variousfullyandlocallyconnectednetwork isfurtherinvestigatedinSection2.3.4.Inthissectionwewilldiscussthe thenumber<strong>of</strong>weights,byforcingsome<strong>of</strong>theweightstobeequal.This lessweightsthanthefullyconnectednets.Wecanevenfurtherreduce<br />
leftside<strong>of</strong>Figure18(inthisgureonlytheconnectionstothersthidden 2.3.1Fullyconnectednets nodearedrawn).Allnodes<strong>of</strong>onelayerareconnectedtoallnodes<strong>of</strong>the Acommonnetworkarchitectureisafullyconnectednetasshownonthe<br />
muchfreedomanditwillnotgeneralize.Morehiddenlayerscanbeadded previouslayer.Thenumber<strong>of</strong>nodes<strong>of</strong>thehiddenlayerisveryimportant. theproblem.Butitshouldalsonotbetoolarge,otherwisethenethastoo toimproveonthetrainingandgeneralizationresults.Usuallyonehidden Thisnumbershouldnotbetoosmall,otherwisethenetisnotabletolearn<br />
<strong>of</strong>theinput. layerisenoughtolearnaproblem,butsometimesanextralayerhelpsto combinethefeaturesfoundbytherstlayer.Itprovidesamoreglobalview
28 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />
Theinputhasastronglocalstructure,soitsimpliestheproblemby<strong>using</strong> Figure18:Fullyconnected(left)andlocallyconnected(right)neuralnets.<br />
Rt<br />
Rt<br />
Rxo<br />
Rxo<br />
onlyseesapart<strong>of</strong>theinputandnotalltheinputsasinthefullyconnected 2.3.2Locallyconnectednets so-calledreceptiveelds.Itmaybeeasierfortheneuralnetifaneuron<br />
dxo<br />
dxo<br />
nets.Theneuronspecializedonitspart<strong>of</strong>theinputandcanbeusedasa localfeaturedetector.Weaddanextrahiddenlayerinordertocombine thelocalfeaturesproperly.Thepart<strong>of</strong>theinputaneuronisconnected to,iscalledareceptiveeld.Usuallyallreceptiveeldshavethesamesize Theweightkernel<strong>of</strong>therstreceptiveeldthatisconnectedtotherst andareonlyshiftedinspace(ortime)withaxedstep(xedoverlap). neuronisshowninFigure19.Wecanconstraintheweightkernelsforthe theneuralnet).Thedecrease<strong>of</strong>freedommightimprovethegeneralization variousreceptiveeldstobethesame,thisiscalledweightsharinganditis ability<strong>of</strong>thenet,butifthefreedomisreducedtoomuch,thenetoverall usedintheconvolutionalnetworks.Theadvantage<strong>of</strong>weightsharingisthe decreaseinthenumber<strong>of</strong>weightsandfreedom(andthereforecomplexity<strong>of</strong> performancemaydecrease.Themotivationforweightsharingisthatwe expectthataparticularmeaningfulfeaturecanoccuratdierenttimes(or locations)intheinput.Anexample<strong>of</strong>alocallyconnectednetisshownin<br />
Inourmodelswehavenodippinglayersandnodeviatedboreholes.In Figure18(right). 2.3.3Symmetryconstraints<br />
foracertainweightvectorw.Themirrorimage<strong>of</strong>x=(x1;x2;:::;xN)is neuralnet(inahiddennodeatthersthiddenlayer)toasignalxisf(x;w), response.Thisshouldalsoholdfortheneuralnet.Theresponse<strong>of</strong>the asignalanditsmirrorimage,asshowninFigure20,givethesametool thiscasethetoolreadingsareassumedtobesymmetric.Thismeansthat
2.3Architectureconstraints 29<br />
W<br />
11<br />
W<br />
12<br />
W<br />
31<br />
W<br />
21<br />
W<br />
22<br />
W<br />
32<br />
W<br />
31<br />
W<br />
32<br />
W<br />
33<br />
weight kernel for<br />
first receptive field<br />
n<br />
x0=(xN;:::;x2;x1).Theresponse<strong>of</strong>theneuralnettothisinputisf(x0;w). Figure19:Receptiveeldswhichoverlap13.<br />
m<br />
first receptive field<br />
first field shifted down 2/3<br />
first field shifted right 2/3<br />
Wewillnowinvestigatewhattheimplicationsareforthearchitectures<strong>of</strong> thenetswhenweforcef(x;w)f(x0;w),given Figure20:Adiscretizedinputsignalanditsmirrorimage.<br />
x1 x2 x3 . . . xn xn . . . x3 x2 x1<br />
foranyhiddennodejinthersthiddenlayer,theindexjisomitted. Forthefullyconnectednetsthefollowingequationshouldhold Here,jstandsforhiddennodej.Sincethefollowingequationsshouldhold<br />
8x2
30 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />
Thisisonlytruewhenwi=wN+1i.Thisconstraintcanbebuildinto weightscomingfrominputnodeN+1i.Theseconstraintsareshownin thenetbyforcingtheweightscomingfrominputnodeitobeequaltothe<br />
Figure21:Symmetryconstraintsonfullyconnectednet.<br />
eldsareconstrainedsymmetrically(receptiveeldfisconstrainedtoeld Ff+1forFelds). Figure21.Thesameconstraintscanbeusedforthelocallyconnectednet Figure22:Symmetryconstraintsonlocallyconnectednet.Thereceptive<br />
symmetricallyconstrained.Thismeansthatreceptiveeldfisconstrained toreceptiveeldFf+1,whereFindicatesthetotalnumber<strong>of</strong>receptive asshowninFigure22.Inthelocallyconnectednetsthereceptiveeldsare elds.Theeldsarealsointernallyconstrainedsymmetricallyasshownin Figure22. constraints.Thewaveletvector<strong>of</strong>aninputx=(x1;x2;:::;xN)(N=2j)on Forthenetsthataretrainedonwaveletcoecients,wecanalsondsome
2.3Architectureconstraints 31<br />
Figure23:Symmetryconstraintsonwaveletnets.Thecoecientsareconstrainedperdetaillevel.<br />
acertainlevelisconstructedby L(x)=k0B@x1+x2<br />
0 1<br />
x3+x4<br />
2<br />
level 3<br />
xN1+xN . 1 CA H(x)=k0B@x1x2<br />
ThecoecientsonthislevelcomefromH(x).Thecoecientsforthemirror xN1xN x3x4 . 1 CA (33)<br />
image<strong>of</strong>xarecalculatedby L(x0)=k0 B @xN+xN1 x2+x1 x4+x3 . 1CA H(x0)=k0 B @xNxN1 x2x1 x4x3 . 1CA Thecoecientsonhigherlevelsarecalculatedrecursivelyonthevector (34)<br />
signalx.Forthe2jwaveletcoecientsonlevelj>0thefollowingequation H(x0)shouldbethesameoneverydetaillevelandforeverypossibleinput shouldholdforanywaveletvectory=H(x)andy0=H(x0) <strong>of</strong>L(x).Theresponse<strong>of</strong>thenettoH(x)andtheresponse<strong>of</strong>thenetto constructedbyL(x).OneverylevelwendthatL(x0)isthemirrorimage<br />
8y2
32 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />
describesthedierencebetweentheaverageoverthersthalf<strong>of</strong>thewindow<br />
networkisshowninFigure23. positiveornegative. Onleveljwendw2j+i=w2j+1ifori=0;1;:::;2j1.Theresulting andtheaverageoverthesecondhalf<strong>of</strong>thewindow.Thetoolreadingsshould besymmetrics<strong>of</strong>orthetoolitdoesnotmatterwhetherthisdierenceis<br />
Aneuronthatislocallyconnectedextractslocalfeatures.Sometimeswe featuresthemselves(forexampleincharacterrecognition). 2.3.4Convolutionalnetwork arenotconcernedaboutthelocation<strong>of</strong>thesefeatures,butonlyaboutthe<br />
convolution subsampling convolution subsampling convolution<br />
input<br />
output<br />
neuroninthersthiddenlayertodetectaspecicfeatureatthelocation<strong>of</strong> thereceptiveeld.Wecanusethesameset<strong>of</strong>weightsforallthereceptive Theset<strong>of</strong>weights,w,belongingtoareceptiveeld,makesitpossiblefora Figure24:Convolutionalnetwork.<br />
map, all neurons<br />
this map share their<br />
wasrstonlydetectedbythersthiddenneuron,anywhereinthetotal elds.Thisenablesthersthiddenlayertodetectthatspecicfeature,that<br />
input.Theneuronsinthersthiddenlayerthatusethisset<strong>of</strong>weightsto <strong>of</strong>anumber<strong>of</strong>neuronsthatmakeuse<strong>of</strong>thesameset<strong>of</strong>weights(weight detectaspecicfeature,arecalledafeaturemap.<br />
kernel.Theconvolution,performedbyonefeaturemap,isdenedbythe sharing).Thisset<strong>of</strong>weights,asshowninFigure19,isusedasaconvolution mapstomakeitpossibletodetectmorefeatures.Eachfeaturemapconsists Thehiddenlayerisnowabletodetectonefeature.Wecanaddmorefeature<br />
sum yj=k+n1 Xkl+m1 Xlwjklxkl (36)
2.3Architectureconstraints 33<br />
wherexklistheinputpixelatlocation(k;l)andwjklistheweightbetween thisinputpixelandneuronj.Theindiceskandlindicatetheleftupper corner<strong>of</strong>thereceptiveeldandnmindicatethesize<strong>of</strong>thereceptiveeld. ceptiveeldsandsharedweights.Theoverlap<strong>of</strong>thereceptiveeldsis subsamplinglayersareliketheconvolutionlayers:theyalsomakeuse<strong>of</strong>re-<br />
Theconvolutionlayersarealternatedbyso-calledsubsamplinglayers.These<br />
isreduced.Duetothisreductioninresolution,thislayerprovidessome allatthesubsamplinglayers.(Whenthereceptiveeldsdonotoverlap maximalthisisusuallycalledsubsampling).Inthesubsamplinglayerthe spatialresolution<strong>of</strong>thefeaturemaps,generatedbytheconvolutionlayers, maximal(replacement<strong>of</strong>onepixel)attheconvolutionlayersandnotat<br />
degree<strong>of</strong>translationalandrotationalinvariance. Thistype<strong>of</strong>networkiscalledaconvolutionalnetworkandisshownin Figure24.<br />
Theprinciplesaremoreorlessthesame.Atimedelayednetworkisusedin recognition.Theone-dimensionalversioniscalledatimedelayednetwork. 2.3.5TimeDelayednetwork<br />
applicationslikespeechrecognition(Waibel,Hanazawa,Hinton,Shikano& Convolutionalnetworksareusedintwo-dimensionalproblems,likecharacter Lang1989). Theideaisthattheresponseatacertainpoint<strong>of</strong>time(ordepth)depends onpreviousinputswithacertaindelay(hencethenametimedelayed).In<br />
featuresinthenextconvolutionlayer.Thisiscalledbi-pyramidalscaling bytherstlayerispartiallycompensatedbyanincreaseinthenumber<strong>of</strong> Thisisbecausewedonotwantthetranslationalandrotationalinvariance, thatisprovidedbythesubsamplinglayers.Theloss<strong>of</strong>timeresolution thistype<strong>of</strong>nettherearenosubsamplinglayers,onlyconvolutionlayers.<br />
(Guyon1991).Thearchitectureforatypicaltimedelayednetworkisshown inFigure25.<br />
number <strong>of</strong><br />
feature<br />
maps<br />
(copies)<br />
features in time<br />
Figure25:Timedelayedneuralnetwork.<br />
feature map<br />
(shared weights within map)<br />
features<br />
(or variables)<br />
receptive field
34 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />
convolutionalnetworksandthetimedelayednetworks.Theideabehindthis 2.3.6Convolutional-regressionnetwork Thedesign<strong>of</strong>theconvolutional-regressionnetworkisbaseduponboththe<br />
copies<br />
number <strong>of</strong><br />
feature<br />
maps<br />
feature map<br />
(shared weights within map)<br />
features<br />
type<strong>of</strong>networkisthatthenetworkoutputdependsonanumber<strong>of</strong>inputs<br />
(or variables)<br />
aroundthepoint<strong>of</strong>interest.Soitdependsoninputsaboveandbelowthe loggingpoint.Forthisnetweusetheinputrepresentationproducedbythe Figure26:Convolutional-regressionnetwork.<br />
features time<br />
(uniform)discretizedslidingwindowapproach.<br />
receptive field<br />
thepreviouslayer.Allotherlayers<strong>of</strong>theconvolutional-regressionnetwork nalconvolutionalnetworks,butincontrastwiththetime-delayednetworks. Inthelatterthefeaturemapsarealwaysconnectedtoallthefeatures<strong>of</strong> some<strong>of</strong>thefeatures(variables)intheinputlayer.Thisisliketheorigi-<br />
Thefeaturemapsinthersthiddenlayercanbeconnectedtoalloronly<br />
information<strong>of</strong>theinput.ThearchitectureisshowninFigure26.Inthisexamplethefeaturemapsareconnectedtoonlyanumber<strong>of</strong>theinputfeatures.<br />
arefullyconnected.Sothisnetonlycontainsoneconvolutionlayer(and thisisdierentfromboththeconvolutionalandtime-delayedarchitectures). Weonlyuseoneconvolutionlayer,becausewewanttoretainthespatial<br />
weights)thatisneededissmall(thenumber<strong>of</strong>hiddennodesisequaltothe number<strong>of</strong>receptiveelds).Adisadvantageistheloss<strong>of</strong>spatialresolution. overconvolutionisthatthenumber<strong>of</strong>hiddennodes(andthenumber<strong>of</strong> thisisnotaconvolution,butsubsampling.Theadvantage<strong>of</strong>subsampling Thereceptiveeldsintherstlayerdonotoverlapmaximal.Soactually
2.4Errorfunction 35<br />
Afeaturemapfromthersthiddenlayerdetectsafeatureintheinput.The<br />
2.4Errorfunction hiddenlayer. secondhiddenlayerdeterminestheimportance<strong>of</strong>thelocation<strong>of</strong>thatfeature.Sothespatialinformationispreservedinthesecond(fullyconnected)<br />
Aneuralnetworkcanminimalizeanyerrorfunction.Themostcommon trainingset errorfunctionusedisthesumsquareerror,whichis,forPpatterns<strong>of</strong>the<br />
patternp. Here,dpisthedesiredoutput<strong>of</strong>patternpandapistheactualoutput<strong>of</strong> ENet=PXp=1(dpap)2 (37)<br />
Error<br />
10<br />
Relative error<br />
Network error<br />
8<br />
6<br />
4<br />
Inthisprojecttheperformance<strong>of</strong>theneuralnetismeasuredintherelative<br />
2<br />
errorbetweenthedesiredandtheactualoutput,averagedoverPpatterns Figure27:Thenetworkandrelativeerrorforxedproportionsd=a.<br />
0<br />
0 1 2 3 4 5 6 7 8 9 10<br />
ERel=vut1PPXp=1 dpap dp!2 (38)<br />
a/d<br />
Theneuralneterrorforonespecicpatternpcanbewrittenas ENet(p)=d2p 1ap dp!2 (39)
36 2.INPUTREPRESENTATIONANDARCHITECTUREDESIGN<br />
patternswithahighdesiredoutput.Thisisnotwhatwewant,becausea smalldesiredoutputismorelikelytohaveahighrelativeerrorthanahigh errorwhenthedesiredoutputislarge.Thenetemphasizesthelearning<strong>of</strong> Thismeansthatforthesameproportiona=d,apatternpproducesahigher<br />
Beforepresentingthepatternstothenet,wehavescaledtheinputandthe targetvalue.ThuswepreferredtheerrorERelasgivenineq.38,which output.Thedesiredandactualoutputsarescaledby overcomesthisdisadvantage.<br />
followingerrorENet=PXp=1ln(dp) Theconsequence<strong>of</strong>thisscalingisthatthenetworkisminimalizingthe t0=ln(t) (40)<br />
=12PXpln2dp ln(a) 2<br />
ap (42) (41)<br />
whichisalmostthesameasminimalizingtherelativeerror,especiallywhen relativeerrorERelaredrawninFigure27. givethesameerrorsignal,asshownbytherelativeerrorinFigure27.The theproportiond=a.Forxedproportionsd=athenetworkerrorENetand Anoverestimation(a=d+k)andanunderestimation(a=dk)must theproportiond=aisalmostequalto1.Actuallythenetisminimalizing<br />
overa=d+k,becausetheformerproducesalowererrorsignal. neuralnet,withtheerrorfunctionasmentionedineq.41,favoursa=dk<br />
wewanttominimalizetherelativeerror.Whenwehadusedanotherscalingmethod(notlogarithmic),thenetworkerrorfunctionshouldhavebeen<br />
Itisveryimportanttochooseanappropriateerrorcriterion.Inthisproject targetvalues,whichisalsothecasefortherelativeerror. Itdoes,however,emphasizethelearning<strong>of</strong>smalltargetvaluesoverlarger<br />
adaptedforbetterperformance.Withthisscalinghoweverthesumsquare errorfunctionsuces.
measuredresistivityisheavilyaectedbytheinvasionresistivity(Rxo),althoughthisdependsontheinvasionradius(dxo).Beforeinvestigatingthis<br />
<strong>Forward</strong>modellinginthepresence<strong>of</strong>mudinvasionisverycomplex.The 3<strong>Forward</strong>modellingwithoutmudinvasion<br />
weonlyhaveonevariable(Rt).Theboreholeradiusanddrillinguidresistivityarexedto4.25inchand0.05m,respectively.Withonlyone<br />
complexsituation,werstlookatthecasewithoutmudinvasion.Here variabletheinputspace<strong>of</strong>earthmodelsisnottoolarge,whichallowsusto<br />
Wehavetakensmallbedsizesthatrangebetween1and5feet.Each wecanusethisinthecasewithinvasion. rstexperimentwiththeinputrepresentationandthenetworkarchitecture.<br />
earthmodelconsists<strong>of</strong>asmalllog<strong>of</strong>50feetlongand15beds,withatotal Whenwehavefoundthemostappropriaterepresentationandarchitecture,<br />
summarizedinTable2. <strong>of</strong>47models(2350feet)fortrainingand33models(1650feet)fortesting. Thetrueresistivityrangesbetween1and70m.Theseparametersare Fortheexperimentationwiththeinputrepresentationandnetworkarchitecturewehaveusedasmalltrainingset<strong>of</strong>4earthmodelsandanequallysized<br />
testset<strong>of</strong>4models.Forthesesmalltestsnoabsoluteerrorsareincluded, theaveragerelativeerror<strong>of</strong>thetrainingsetandthetestset.Anetperforms sincetheyareonlyusedforcomparison.Theperformanceismeasuredin wellwhenbotherrorsaresmallandcomparable. Table2:Parametersforearthmodelswithoutinvasion. Fixedparameters Boreholeradius Drillinguidresistivity Variables Bedsize 4.25inch<br />
Trueresistivity 1,2,3,4or5feet 1,2,3,...,70m 0.05m<br />
Data Trainingset Testset Permodel 251examples 47models 33models 50feet 15beds<br />
37
38 3.FORWARDMODELLINGWITHOUTMUDINVASION<br />
Theinputandoutputareapproximatelyscaledtotherange[1;+1](see 3.1Experimentingwithdierentscalingmethods Section2.1.3).Thisisdonebynormalization<strong>of</strong>theinputandoutput<br />
whereisthemeanandisthestandarddeviation<strong>of</strong>thevariablezonthe z0=z<br />
logarithmicscaling.Inthatcasethemeanandstandarddeviation<strong>of</strong>ln(z) trainingset.Thistype<strong>of</strong>scalingcanalsobedoneincombinationwitha (43)<br />
isused.Themeanandstandarddeviationsfortheinputandoutputare giveninthefollowingtable logarithmicscaledRt logarithmicscaledLLdorLLs3.000.83 29.0016.67 30.0016.67 3.300.80 <br />
applied,sothenormalizationonitselfisnotsucient.Whennologarithmic bestresults.Theworstresultsarefoundwhennologarithmicscalingwas thatthelogarithmicscalingincombinationwiththenormalizationgavethe andonehiddenlayer<strong>of</strong>5nodeson1004examples(4models)andfound Wetestedafullyconnectednet<strong>of</strong>25inputs(slidingwindow<strong>of</strong>5feet)<br />
errorandaminimalization<strong>of</strong>theabsoluteerrordoesnotnecessarymeana problem.Duetothisscalingtheproportionisminimalized(seeSection2.4), thedesiredandactualoutput.Wemeasuretheperformanceintherelative minimalization<strong>of</strong>therelativeerror.Thelogaritmicscalingovercomesthis scalingwasapplied,theneuralnetisminimizingtheabsoluteerrorbetween<br />
whichisalmostthesameasminimalizingtherelativeerror. Wewillusethiscombinedscalingmethodintheothertests.<br />
experimentsaredoneforawindowsize<strong>of</strong>5feet.Thetestsaredoneforboth 3.2Experimentingwiththenetworkarchitecture Nowwestartedexperimentingwiththenetworkarchitecture.Thefollowing theDeep-andtheShallow-logwith4trainingmodels(1004examples). Variation<strong>of</strong>number<strong>of</strong>hiddennodes(5,10,15,20,25).Wefound Variation<strong>of</strong>number<strong>of</strong>hiddenlayers(1or2).Twolayers<strong>of</strong>both15 thatnetswith15nodesandmoregavesimilarresults. thatincreasingthenumber<strong>of</strong>nodesupto15improvetheresults,but nodesoronehiddenlayer<strong>of</strong>15nodesgavecomparableresults.
3.3Experimentingwiththesize<strong>of</strong>theslidingwindow 39<br />
Whentheresultsarecomparablewechooseforthenetwiththefewest Addingconnections<strong>of</strong>theinputlayerdirectlytotheoutputlayer(25<br />
number<strong>of</strong>weights.Thesenetsmostlikelyhavethebestgeneralizationand thenormalfullyconnectednetwerecomparable. extraconnections).Againtheresultsfortheseextraconnectionsand<br />
thesmallesttrainingtime.Thebestarchitecturefoundbytheseexperiments<br />
Thescalingmethodandthenetworkarchitecturearexed,onlythewindow 3.3Experimentingwiththesize<strong>of</strong>theslidingwindow isanormalfullyconnectednet,withonehiddenlayer<strong>of</strong>15nodes.<br />
sizeisvariedinthefollowingtests.Wetriedwindowsizes<strong>of</strong>5,10,15and 30feet.Thesmallestwindowsizewecantryis5feet(seeSection2.1.1). Againthenetsaretrainedon4models(1004examples),onboththeDeepandtheShallow-log.Inbothcases,awindowsize<strong>of</strong>15feetgivesthebest<br />
thisdoesnotoutweighthelongertrainingtimeduetothishighnumber<strong>of</strong> weights(2281versus1156). results.Althoughtheresultsareslightlybetterforawindowsize<strong>of</strong>30feet, TheShallow-logapproximationisbetterthantheDeep-logforthesame windowsize.Wealreadyexpectedthis,becausetheDeepLaterolog\sees" alargerpart<strong>of</strong>theformationandthereforeneedsalargerwindowsize.This<br />
Thearchitecturewefoundbytheprevioustestsisafullyconnectedneural 3.4Summaryandresults wasalreadyshowninFigure3inSection1.1.1.<br />
netwithonehiddenlayer<strong>of</strong>15nodesandawindowsize<strong>of</strong>15feet(75inputs).Thisnetistrainedon47modelswithatotallength<strong>of</strong>2350feetand<br />
testedon33modelswithatotallength<strong>of</strong>1650feet.<br />
Table3:Averagerelativeerrorforearthmodelswithoutinvasion(1). Shallow-logTrainingset1.7% Deep-log Trainingset5.1% Testset 2.2% 6.0%
40 3.FORWARDMODELLINGWITHOUTMUDINVASION<br />
proximation<strong>of</strong>theShallow-log.Thegeneralizationability<strong>of</strong>thenet,that AsshowninTable3,theaveragerelativeerrorisbelow5%fortheap-<br />
istrainedontheShallow-log,isquitegood(thetesterrorandthetraining relativelylargecontributiontotheaveragerelativeerror.Therefore,wealso errorarecomparable). However,theaveragerelativeerrormaynotbeanappropriateerrorcriterion,sinceitispossiblethatafewpointswithahighrelativeerrorhavea<br />
calculatethepercentage<strong>of</strong>theoutputthathasarelativeerrorbelow5% (called\correct")asshowninTable4.Fromthistablewendthatonly Table4:Performanceonearthmodelswithoutinvasion(1). Shallow-logTrainingset98%correct Deep-log Trainingset62%correct Testset 96%correct<br />
4%<strong>of</strong>theapproximation<strong>of</strong>theShallow-loghasarelativeerrorabove5%. thelearningandgeneralizationability<strong>of</strong>thenetarequitegood. Andthisisveryclosetotheerroronthetrainingset.Therefore,wecansay 61%correct<br />
<strong>of</strong>theshoulderbeds,whichislesspr<strong>of</strong>oundinthecase<strong>of</strong>theShallow-log Theapproximation<strong>of</strong>theDeep-log,however,causesmoreproblems.This sistivityprole,incontrasttotheShallow-log.Thisiscausedbytheeect (seeSection1.1.2). isshowninFigure28.TheDeep-logdierssubstantiallyfromthetruere-<br />
TheoutputtheneuralnetgivesforthismodelisshowninFigure29.In thisgureoneseesthattheoutputgivenbythenetlooksmoreliketheresistivityprole(liketheShallow-log)andthereforehasquitealargerelative<br />
errorwithitstargetvalue(theDeep-log).Animprovementcouldbemade Weconcludethatourinputrepresentation(slidingwindowandscaling)is by<strong>using</strong>moremodelssimilartothisoneinthetrainingset.<br />
willabandonthefullyconnectednetsandlookformorecomplexarchitecturestoimproveontheresultsandtohandleinvasion.<br />
appropriatefortheproblem.Afullyconnectedneuralnetwithonehidden generalizationoverdataithasnotseenbefore.Inthefollowingchapterwe layerissucienttolearntheinput-outputmappingandtoprovidegood
3.4Summaryandresults 41<br />
Ohm meter<br />
80<br />
70<br />
True resistivity<br />
Deep-log<br />
Shallow-log<br />
60<br />
50<br />
40<br />
30<br />
Figure28:ModelthatcausesdicultiesinapproximatingtheDeep-log.<br />
20<br />
10<br />
0<br />
0 5 10 15 20 25 30 35 40 45 50<br />
depth<br />
Ohm meter<br />
80<br />
70<br />
True resistivity<br />
<strong>Neural</strong> Net<br />
Deep-log<br />
60<br />
50<br />
40<br />
30<br />
Figure29:Response<strong>of</strong>theneuralnettothedicultmodelshownabove.<br />
20<br />
10<br />
0<br />
0 5 10 15 20 25 30 35 40 45 50<br />
depth
42 4.FORWARDMODELLINGWITHMUDINVASION<br />
4<strong>Forward</strong>modellingwithmudinvasion Nowwehavefoundthatitispossibletolearnthemappingbetweenthe earthmodelsandthetoolresponse,wearegoingtolookatamoredicult (andinteresting)problem.Inthenewearthmodelsthelayers<strong>of</strong>theformationareinvadedbythedrillinguid.ThismeanswealsohavetotakeRxo,<br />
boreholeradiusandtheresistivity<strong>of</strong>thedrillinguidarexed.Theinput theinvasionresistivity,anddxo,theinvasionradius,inaccount.Againthe<br />
moreexamplestolearntheproblem. thethreevariablesRt,Rxoanddxo,ismuchlargerthanintheprevious space<strong>of</strong>earthmodels,constructedfromrandomlychosencombinations<strong>of</strong><br />
Werstperformanumber<strong>of</strong>testsonasmalltrainingsetinordertoget anideaabouthowwellthechoseninputrepresentationandarchitecture case,whentherewasonlyonevariable(Rt).Weexpectthenetworkneeds<br />
performonan(alsosmall)testset.<br />
Fixedparameters Boreholeradius Table5:Parametersforearthmodelswithinvasion.<br />
Drillinguidresistivity Variables Bedsize 1,2,3,...,20feet 0.05m 4.25inch<br />
Trueresistivity Invasionresistivity Invasionradius Data 0.5,0.7,...,2.5m 2,3,4,...,71m<br />
Trainingset Validationset A,B,C,D,E,F(M,N,O,P,Q,RandS)models 8,9,....,50inch<br />
Testset Permodel J,KandLmodels G,HandImodels 1000feet<br />
Weusebedsizesbetween1and20feet,whichismorerealisticthaninthe 5000examples 80beds<br />
training,theso-calledvalidationset.Whentheerroronthissetstartsto 19modelsfortraining,testingandvalidationasshowninTable5.During trainingonecalculatestheerroronaset<strong>of</strong>examplesthatisnotusedduring isapproximately1000feetlongandconsists<strong>of</strong>80beds.Thereisatotal<strong>of</strong> previoustests,whereweusedbedsbetween1and5feet.Eachearthmodel<br />
increase,thetrainingisstopped.Inthesmalltestswehavenotuseda
4.1Scaling<strong>of</strong>theparameters 43<br />
validationset.Thetrainingwasstopped,whenthenetworkconvergedtoa<br />
tolearnthanthemappingbetweentheearthmodelsandtheDeep-log.Ifa isminimal,buttheerroronthetestsetisnot.Intheseexperimentsweonly thatthemappingbetweentheearthmodelsandtheShallow-logwaseasier usetheShallow-log.Wehavefoundintheprevioustests,withoutinvasion, minimum.Thismeansthenetsareovertrained:theerroronthetrainingset<br />
(A)<strong>of</strong>4601samplepointsastrainingsetandonemodel(F)<strong>of</strong>4876sample Inthenextsectionssome<strong>of</strong>theresultsaregiveninduplicateinorderto getabettercomparisonbetweenthevarioustests.Wehaveusedonemodel moredicultyinlearningtheDeep-log. netndsitdiculttolearntheShallow-log,weexpectitwouldhaveeven<br />
ThetrueresistivityRtandtheShallow-andDeep-log(LLsandLLd)are 4.1Scaling<strong>of</strong>theparameters pointsastest(generalization)set.<br />
scaledasintheprevioustests.Thescalingmethodswehaveusedhereare summarizedinthefollowingtable VariableDomain Rt Rxo dxo LLd 2,3,...,71 0.5,0.7,...,2.3mnorm mlog+norm Scaling MeanDeviation<br />
LLs 8,9,...,50 inchnorm log+norm29.00 3.30 1.30 1.40 25.00<br />
0.59<br />
Inthefollowingsectionsweusetablestodescribetheperformedtests.The 1.67<br />
followingtableentriesareusedtodescribeatest:<br />
architecture:Adescription<strong>of</strong>thenetperlayerseperatedbya\.". connections:Thetype<strong>of</strong>connections.Thisisfullyorlocallyconnected(fullyconnectedbydefault).<br />
3128forexamplemeanswehave128valuespervariable.Alayer<strong>of</strong> splitintothenumber<strong>of</strong>valuespervariableorfeaturemap.Alayer<strong>of</strong> Foreachlayerthenumber<strong>of</strong>nodesisgiven.Thisnumbercanbe<br />
windowsize:Theusedsize<strong>of</strong>theslidingwindowisgiveninfeet Number<strong>of</strong>weights:Thenumber<strong>of</strong>weightsinthenetisgiven. 627meanswehave6featuremaps<strong>of</strong>27nodeseach.<br />
Samplingperiod:Theusedsamplingperiodisgiveninfeet(default valueis0.2feet). (defaultvalueis25.4feet).
44 4.FORWARDMODELLINGWITHMUDINVASION<br />
Sampling:Thetype<strong>of</strong>sampling.Thiscanbeuniform(defaultvalue), Number<strong>of</strong>epochs:Thenumber<strong>of</strong>epochsthatwasneededtotrain non-uniformornone.<br />
Generalizationerror:TheaveragerelativeerroronmodelF. Trainingerror:TheaveragerelativeerroronmodelA. thenet.<br />
4.2Experimentingwithdierentinputrepresentations<br />
inTest1.Awindowsize<strong>of</strong>25.4feetandasamplingperiod<strong>of</strong>0.2feetgives attheinputrepresentationcomingfromauniformsampledslidingwindow Because<strong>of</strong>theresultsintheexperimentswithoutinvasion,werstlooked Inthismorecomplexsituationwehavetolookatthreevariablespersamplingpoint.Thisresultsinahighnumber<strong>of</strong>inputsfortheneuralnet.<br />
us3128inputs,whichisquitehigh.ButwealsolookedataninputdescriptionbyattributesinTest2asdescribedinSection2.1.2.Theresults<br />
Sincetheslidingwindowrepresentationisgivingbetterresultsthanthe aregiveninTable6.<br />
bad.Thenetisheavilyovertrained,sincetheerroronthetestsetwasonly thenumber<strong>of</strong>weights. Thegeneralizationperformance<strong>of</strong>theattributesinputrepresentationisvery attributesdescription,wewillnowlookformethodstoreducethecomplexity<br />
41.8%after183epochs(trainingerroratthattimewas11.4%). <strong>of</strong>thenet.Thiscanbedonebyreducingthenumber<strong>of</strong>inputsorbyreducing<br />
first sampling point,<br />
inverse distance is 10<br />
0.2 feet<br />
second sampling point,<br />
inverse distance is 5<br />
Contrasts with adjacent bed<br />
Figure30:Smalldierenceininputwithattributesinputrepresentation.<br />
Input at first sampling point Rt Rxo dxo Rt/Rt2 Rxo/Rxo2 dxo/dxo2 Rt-Rt2 Rxo-Rxo2 dxo-dxo2<br />
Input at second sampling point Rt Rxo dxo Rt/Rt2 Rxo/Rxo2 dxo/dxo2 Rt-Rt2 Rxo-Rxo2 dxo-dxo2<br />
True resistivity, invasion resistivity and<br />
invasion radius<br />
Differences adjacent bed<br />
10<br />
5
4.2Experimentingwithdierentinputrepresentations 45<br />
tation.Representation Table6:Comparingdiscretizedslidingwindowagainstattributesrepresen-<br />
architecture number<strong>of</strong>weights windowsize samplingperiod Test1 3128.10.1 3861 Test2<br />
25.4feet 100.15.15.1<br />
inputdescription Uniform DiscretizedslidingAttributes 0.2feet 1771<br />
window 30feet<br />
number<strong>of</strong>epochs trainingerror generalizationerror70.0% 3060 3.1% 3484 (10beds) 4.2% >100%<br />
Table7:Comparinguniformsamplingmethodswithdierentsamplingperiods.<br />
Sampling1 architecture number<strong>of</strong>weights windowsize samplingperiod Test1 3128.10.13100.10.1 3861 25.4feet Test4<br />
number<strong>of</strong>epochs 0.2feet 3021<br />
trainingerror 3060 29.7feet<br />
generalizationerror70.0% 3.1% 0.3feet 3634 2.9% 67.3%<br />
Table8:Comparinguniformandnon-uniformsamplingmethods. Sampling2 architecture number<strong>of</strong>weights windowsize samplingperiod Test3 3128.10.10.1364.15.15.1 25.4feet 3971 29.4feet Test5<br />
Uniform 0.2feet 3151<br />
number<strong>of</strong>epochs trainingerror generalizationerror65.2% 6429 1.1% 46.4% Non-uniform 4404 1.0%
46 4.FORWARDMODELLINGWITHMUDINVASION<br />
eachbeddiersinthosecases,asshowninFigure30.Theinversedistance becomesthemostimportantvalueintheinput.Thiscanalsobeseenfrom theweights.Mostweightsarequitesmall,rangingfrom-0.5to0.5,butthe fedwithanumber<strong>of</strong>almostsimilarexamples.Onlytheinversedistancefor Apossiblecause<strong>of</strong>thebadperformanceontheattributesisthatthenetis<br />
weightsbelongingtotheconnectionsfromtheinversedistance-inputsto thehiddenlayerarelarge(1.5). isverydicult.Thenetworkisactuallypushedinsomekind<strong>of</strong>direction Itisimportanttochooseappropriateattributes,butndingtheseattributes by<strong>using</strong>attributes.Thiscanmakethelearningeasier,butitcanjustas easymakethelearningmoredicult. 4.3Experimentingwithinputreductionmethods<br />
nodesandmaybeeventwohiddenlayers<strong>of</strong>15nodes.Theformergivesus veryhigh.Forexample,awindow<strong>of</strong>29.8feetandasamplingperiod<strong>of</strong> 0.2feetproduces3150inputs.Wewouldliketouseahiddenlayer<strong>of</strong>15 Thenumber<strong>of</strong>inputscomingfromauniformsampledslidingwindowis 6781weightsandthelatter7021weights.Totrainsuchalargenetweneed approximately70000examples(Widrow'sRule<strong>of</strong>Thumbtotakeabout10 timesthenumber<strong>of</strong>freevariables).Eachexampleconsists<strong>of</strong>451values 31570000valuestotrainthisnet.Thislargesetwillslowdownthetraining (3150inputsandonetarget),sowewouldneedtostoreapproximately Itisforthisreasonthatwelookedatseveralwaystoreducethenumber<strong>of</strong> inputsforaxedwindowsize.Thedisadvantage<strong>of</strong>inputreductionisthe andcausesstorageandmemoryproblems. factthatwelooseinformation. Thenumber<strong>of</strong>inputsdependsontheusedwindowsizeandtheusedsamplingperiod.Wecoulduseasmallerslidingwindowthan25.4feet,butthis<br />
isnotanattractiveoption,becausewesuspectthetoolreadingsareaected byawindow<strong>of</strong>atleastthesize<strong>of</strong>thetool(seeSection2.1.1). 3100inputs,instead<strong>of</strong>the3150inputswitha0.2feetsamplingperiod (Test4)instead<strong>of</strong>0.2feet.Inaslidingwindow<strong>of</strong>29.7feetthisresultsin inaslidingwindow<strong>of</strong>29.8feet. Anotheroptionistouseacoarsersamplingperiod,forexample0.3feet Anotherapproachistouseanon-uniformsamplingmethod.Thisapproach<br />
4.3.1Usingdierentsamplingmethods<br />
samenetasinTest1,butwithtwohiddenlayers.Theresultsforthese only364inputsperwindow(instead<strong>of</strong>3150).InTest3weusethe uniformsamplingmethodinTest5asshowninFigure31,whichresultsin isbasedonthefactthatthephysicaltoolreceivesmost<strong>of</strong>itsinformation fromthecenterandlessfromthesides.Wehavechosenanheuristicnon-
4.3Experimentingwithinputreductionmethods 47<br />
<strong>of</strong> 29.4 feet<br />
testsareshowninTable7andTable8. Figure31:Non-uniformsamplinginslidingwindow.<br />
7.2 feet uniform sampled every 0.8 feet<br />
Whenauniformsamplingmethodisused,ititpossibletouseacoarsersamplingperiod(0.3feetinstead<strong>of</strong>0.2feet),becausethetrainingandtesting<br />
6.0 feet sampled every 0.4 feet<br />
3.0 sampled every 0.2 resultsaresimilar.Thebedboundariesaredescribedlessaccuratelywhen acoarsersamplingperiodisused.Butfromthesetestswendthatitdoes notinuencetheperformanceinanegativeway. becausethecenter<strong>of</strong>thewindowissampledwiththesamesamplingperiod asintheoriginalinput.Thenetthatistrainedonthisinputrepresentation Withthenon-uniformsamplingmethodweonlylooseaccuracyattheedges,<br />
odstoreducetheinputs. performsbetterthanthenetthatwastrainedontheoriginalinput. 4.3.2Reducingtheinputbyprojectiontoprincipalcomponents Bothmethods,coarserandnon-uniformsampling,areappropriateasmeth-<br />
WeusetheprincipalcomponentanalysisasdescribedinSection2.2.1.The eigenvectorsarecalculatedforaninputleconsisting<strong>of</strong>sixmodels.The modelsarenotalikeinternally,soitisbettertouseaset<strong>of</strong>models.Otherwisetheeigenvectorswillbeappropriateforonemodel,butnotforanother.<br />
rescaledwiththeinversenormalizationscaling<strong>of</strong>equation9,beforethe Wefoundthatitdoesnotmatterwhetheryouusesixmodelsormore,so wehavechosenforthemodelsA,M,N,P,QandR(thesemodelsarevery dissimilar).<br />
iseverywhereequaltothemean<strong>of</strong>theinputvalues.Thisrescalingisnot not100%.Thescaledvectorconsists<strong>of</strong>onlyzero's,buttherescaledvector LOIiscalculated.Duetothisscaling,theloss<strong>of</strong>informationforM=0is Bothvectors(originalN-dimensionalandprojectedM-dimensional)are<br />
absolutelynecessary,butwewantedtocalculatetheloss<strong>of</strong>informationon theoriginalinput. InTest7theoriginalinputvectorconsists<strong>of</strong>364inputs,constructedby
48 4.FORWARDMODELLINGWITHMUDINVASION<br />
Loss <strong>of</strong> information<br />
in percent<br />
50<br />
45<br />
40<br />
LOI on true resistivity Rt<br />
LOI on invasion resistivity Rxo<br />
LOI on invasion radius dxo<br />
35<br />
30<br />
25<br />
20<br />
15<br />
Figure32:Loss<strong>of</strong>informationpervariableforTest6.Originalinputconsists <strong>of</strong>3128inputs,comingfromauniformsampledslidingwindow.<br />
10<br />
5<br />
0<br />
0 8 16 24 40 48 56 64 72 80 88 96 104 112 120 128<br />
Number principal comonents used (M)<br />
principalcomponents. Table9:Comparinguniformsampledinputwithoutandwithprojectionto PCA1 architecture number<strong>of</strong>weights Test3 3128.10.10.1332.10.10.1 No 3971 Test6 1091 Yes<br />
number<strong>of</strong>epochs LOI7.4%onRt<br />
trainingerror 6429 LOI4.7%onRxo<br />
generalizationerror65.2% 1.1% 52.3% LOI4.4%ondxo 5193 1.7%
4.3Experimentingwithinputreductionmethods 49<br />
Loss <strong>of</strong> information<br />
in percent<br />
50<br />
45<br />
40<br />
LOI on true resistivity Rt<br />
LOI on invasion resistivity Rxo<br />
LOI on invasion radius dxo<br />
35<br />
30<br />
25<br />
20<br />
15<br />
Figure33:Loss<strong>of</strong>informationpervariableforTest7.Originalinputconsists <strong>of</strong>364inputs,comingfromanon-uniformsampledslidingwindow.<br />
10<br />
5<br />
0<br />
0 8 16 24 32 40 48 56 64<br />
Number principal comonents used (M)<br />
tiontoprincipalcomponents. Table10:Comparingnon-uniformsampledinputwithoutandwithprojec-<br />
PCA2 architecture number<strong>of</strong>weights windowsize samplingperiod Test5 364.15.15.1332.15.15.1 29.4feet 3151 Test7<br />
PCA Non-UniformNon-Uniform<br />
1711 29.4feet Yes<br />
number<strong>of</strong>epochs LOI7.2%onRt<br />
trainingerror 4404 LOI4.2%onRxo<br />
generalizationerror46.4% 1.0% LOI3.9%ondxo 3896 1.4% 45.6%
50 4.FORWARDMODELLINGWITHMUDINVASION<br />
TheLOI,forbothtests,iscalculatedforP=1520patternscomingfrom modelsJ,OandS. values<strong>of</strong>MisshowninFigure33. anon-uniformsampledslidingwindowasinTest5.TheLOIforvarious<br />
forthetrainingandtestset. Inthefollowingtables,Table9andTable10,theLOIisgivenpervariable<br />
weightsarereducedwith73%and46%respectively. weachieveaninputreduction<strong>of</strong>75%andinthesecondtest50%.The Inbothtestswendthatthetrainingandgeneralizationerrorforthetests analysisisagoodmethodtoreducethenumber<strong>of</strong>inputs.Inthersttest withandwithoutPCAarecomparable.Thismeanstheprincipalcomponent<br />
4.3.3Reducingtheinputsbyremovingwaveletcoecients originalinputs.Wedonotknowhowmuchcoecientscanberemovedon Inthefollowingtestsweusewaveletcoecientsasinputsinstead<strong>of</strong>the thenerdetaillevelssothatnottoomuchinformationislost.Therefore,<br />
Therstinterestingaspectisthatthenetlearnsbetterwhencoecientsare TheresultsaregiveninTable11andTable12. thecoecientsonthenestdetaillevelandsomeonhigherdetaillevels. coecientsareremovedatallandinthelasttest,Test11,weremoveall werunseveraltestswithdierentreductions.Inthersttest,Test8,no<br />
<strong>of</strong>thebadperformance<strong>of</strong>thenetinTest8. inputnodejis0,thereisnoweightcorrection.Thisisprobablythecause localgradientandmostimportantontheinputsignal<strong>of</strong>nodej.Sowhen latedasindicatedbyequation7.Itdependsonthelearningparameter,the removed.Duringtrainingtheweightcorrectionforahiddennodeiscalcu-<br />
becausethedierencebetweenthersthalfandthesecondhalf<strong>of</strong>(apart Thewaveletcoecientsare0whenthereisnobedtransitionatthatspot, becausethereareonlyafewbedboundariesinthewindow.Sometimesa <strong>of</strong>)thewindowis0.Most<strong>of</strong>thecoecientsonhigherdetaillevelsarezero, bedboundaryisnotevendetectedasshowninFigure34. Onthenestdetaillevelthisproblemisquitesevere,becausethesecoecientsonlyseeaverysmallpart<strong>of</strong>theinputandthissmallpartisnotlikely<br />
hiddennodedoesnot\learn"fromthiscoecient. fromthiscoecienttothehiddennodeisnotupdated.Inotherwords,the tocontainabedboundary.Wheneverthebedboundaryisnotdetectedor whenthereisnobedboundaryatall,thecoecientsarezeroandtheweight
4.3Experimentingwithinputreductionmethods 51<br />
Table11:Comparinginputrepresentationswithdierentnumber<strong>of</strong>wavelet coecients(1). Haar1 architecture number<strong>of</strong>weights coecientreductionnone Test8 3128.10.1368.15.1 3861 Test9<br />
number<strong>of</strong>epochs 3091<br />
trainingerror 2125 220onlevel6<br />
generalizationerror63.5% 4.1% 210onlevel5 4303 0.8% 51.5%<br />
Table12:Comparinginputrepresentationswithdierentnumber<strong>of</strong>wavelet coecients(2). Haar2 architecture number<strong>of</strong>weights coecientreduction225onlevel6232onlevel6 Test10 348.10.1 2191 212onlevel5212onlevel5 Test11<br />
23onlevel423onlevel4 334.15.1 1561<br />
number<strong>of</strong>epochs trainingerror generalizationerror50.3% 4479 1.0% 56.4% 3848 0.9%
52 4.FORWARDMODELLINGWITHMUDINVASION<br />
b<br />
Part <strong>of</strong> input signal that lies<br />
in sliding window, wavelet<br />
a<br />
coefficients on this level<br />
are (0,(b-a)/2,0,0)<br />
b<br />
a<br />
Input signal after sliding window<br />
is moved 0.2 feet to the right;<br />
wavelet coefficients on this level<br />
are (0,0,0,0)<br />
discretized input values<br />
1 2 3 4 5 6 7 8<br />
Thereareapproximately2bedboundariesperpattern.Notallboundaries Figure34:Notalltransitionsaredetectedbywaveletcoecients.<br />
aredetectedbythecoecientsonthenestdetaillevel,butwhenitisnot detectedbycoecientciitwillbedetectedbycoecientci1aftermoving<br />
coefficients<br />
theslidingwindow.Soonaverage50%<strong>of</strong>theboundariesisdetectedon thenestdetaillevel.Thetrainingsetconsists<strong>of</strong>4601patterns,sothis meanstheweightcomingfromcoecientcitohiddennodehjisupdated approximately72times(4601patterns,2boundariesperpatternfromwhich 50%isdetectedand64coecients).Ausualtrainingtakesabout2000to<br />
WhenwecomparetheresultstothenettrainedinTest1,weseethatthe thesenetsismuchbetterthantheperformance<strong>of</strong>thenetinTest8. detaillevelandals<strong>of</strong>romhigherlevelsandweseethattheperformance<strong>of</strong> InTest9,10and11weremoveanumber<strong>of</strong>thecoecientsonthenest 3000updatesperweight.<br />
lessweights.Thenumber<strong>of</strong>trainingexamplesinproportiontothenumber forthebettergeneralizationperformance,isthatthenetscontainmuch <strong>of</strong>73%ispossible,withoutreducingthenetperformance.Oneexplanation inputsisreducedfrom3128to368,348and334.Aninputreduction netshavebettertrainingandgeneralizationerrors,althoughthenumber<strong>of</strong><br />
analysis<strong>of</strong>Test6and7. improvementingeneralizationisbetterthanfortheprincipalcomponent andgeneralizationmorethantheoriginaldiscretizedslidingwindow.The possiblethatthenetndsthisinputrepresentationfacilitatesthetraining <strong>of</strong>weightsishigherintheseteststhaninTest1.Butitcouldalsobe
4.4Creatingamorerepresentativetrainingset 53<br />
4.4Creatingamorerepresentativetrainingset Ourtrainingsetisnotveryrepresentative,becausetheexamples<strong>of</strong>adjacent loggingpointslookverysimilar.Thisiscausedbythefactthatweusea<br />
logandsecondlyby<strong>using</strong>\dicult"(non-similar)examples. ited).Therearetwomethodsweemploytoincreasetherepresentativeness <strong>of</strong>thetrainingset:rstlyby<strong>using</strong>acoarsersamplingperiodforthetarget becausewewantasmuchexamplesperlogaspossible(thedatasetislim-<br />
densesamplingperiodforthetargetlog.Weusethisdensesamplingperiod<br />
trainingset<strong>of</strong>thesamesizeasouroriginaltrainingsetweuse15models ducesabout300examples.ThisiswhatwedidinTest12.Tocreatea withthissamplingperiod.Thistrainingsetismorerepresentative,because Whenweuseasamplingperiod<strong>of</strong>3feet,onemodel<strong>of</strong>1000feetonlypro-<br />
thenetwork. itcontainsmoredierentexamples.Weexpectthiswillmakeitmoredif-<br />
Aftertrainingthenet,wehavelookedattheapproximation<strong>of</strong>thenettoan cultforthenettolearntheproblem.Thenetworkhastondamore<br />
arbitrarylog.Wefoundthatthehighestrelativeerrorsoccuraroundareas generalsolutiontotalltheexamples.Thisimprovesthegeneralization<strong>of</strong><br />
wouldbemorerepresentativewhenitcontainedmoreexampleslikethis. Actuallythisissomekind<strong>of</strong>weighing<strong>of</strong>theexamples.Byoeringmore <strong>of</strong>lowresistivityandareaswithhighresistivitycontrasts.Thetrainingset<br />
theotherparts<strong>of</strong>logA(withatotal<strong>of</strong>6036examplesastrainingset).The \dicult"examples,theseexampleswillgetmoreattentionintheminimalizationprocess.InTest13weuseatrainingset<strong>of</strong>4600examples,coming<br />
fromdicultparts<strong>of</strong>logA,PandSand1436coarsersampledexamples<strong>of</strong> resultsforthesetestsaregiveninTable13andTable14. Aswecanseefromthesetests,bothmethodsimprovethegeneralization performance<strong>of</strong>thenet.Thetraining,however,becomesmorecomplicated. dierentexamplesthatlooklikeexamplesthatarefoundinreality. Itisveryimportanttocreateatrainingsetthatcontainsalargenumber<strong>of</strong>
54 4.FORWARDMODELLINGWITHMUDINVASION<br />
Table13:Comparingtrainingset<strong>of</strong>targetlogsandtrainingset<strong>of</strong>coarser sampledtargetlogs. Trainingset1 architecture number<strong>of</strong>weights trainingset number<strong>of</strong>epochs Test1 3128.10.13128.10.1 3861 modelA Test12<br />
trainingerror generalizationerror70.0% 3060 3.1% 3861 mixture<strong>of</strong>15models 3023 5.6% 26.9%<br />
parts<strong>of</strong>targetlogs. Table14:Comparingtrainingset<strong>of</strong>targetlogsandtrainingset<strong>of</strong>dicult Trainingset2 architecture number<strong>of</strong>weights trainingset Test1 3128.10.13128.10.1 3861 Test13<br />
number<strong>of</strong>epochs modelA 3861<br />
trainingerror 3060 dicultexamples<strong>of</strong><br />
generalizationerror70.0% 3.1% modelsA,PandS 1042 9.8% 30.3%
4.5Intermediateresultsinputrepresentations 55<br />
4.5Intermediateresultsinputrepresentations<br />
validationset(14153examples)andthemodelsJ,K,L,M,N,O,P,Q,R Weselectsome<strong>of</strong>thenetsfromthepreviousteststotrainthemonalarge<br />
andSastestset(47485examples). thetrainingsetcontains28331examples.WeusethemodelsG,HandIas modelsA,B,C,D,EandF.Thetargetlogsaresampledevery0.2feet,so trainingset.ThetrainingsetfortheShallow-logisconstructedfromthe<br />
themodelsthenetistrainedon,theothermodelsareusedforvalidation andtesting.Therstnet,Non-uniform,isanetwithanon-uniformsampledinputandPCAreductionfrom364inputsto35inputsforRtand<br />
InFigure35theresultsforthreenetsareshown.Therstsixmodelsare<br />
net,UniformfromTest4,isauniformsampledinputwithawindow<strong>of</strong> some<strong>of</strong>thecoecientsfrom3128inputsto368inputs.Andthethird 29.7feetandasamplingperiod<strong>of</strong>0.3feet,resultingin3100inputs. netwithHaarcoecientsasinputs.Theinputsarereducedbyremoving 20inputsforbothRxoanddxo.Thesecondnet,HaarfromTest9,isa<br />
InFigure35thepercentage<strong>of</strong>thelogthathasarelativeerrorbelow5%is andtestingset. resultsonthetrainingset,buttheyallperformequallyoverthevalidation shown.<br />
Themethodstoreducethenumber<strong>of</strong>inputsworkverywell.Thenetworks AscanbeseeninFigure35,thenon-uniformsampledinputgivesthebest<br />
areabletolearntheproblemevenwhenalargenumber<strong>of</strong>theinputsis removed.Thegeneralizationperformance,however,isnotsucientandwe havealsolostinformation.Inthenextsectionswewillinvestigatecertain architecturalconstraintsinordertoimproveonthegeneralizationperformance.
56 4.FORWARDMODELLINGWITHMUDINVASION<br />
percentage <strong>of</strong> log<br />
that is correct<br />
100<br />
95<br />
90<br />
85<br />
80<br />
75<br />
70<br />
Non-Uniform<br />
Haar<br />
Uniform<br />
65<br />
Figure35:Intermediateresults<strong>of</strong>trainingthreedierentneuralnetson6<br />
60<br />
models.<br />
55<br />
50<br />
45<br />
training validation testing
4.6Experimentingwitharchitectureconstraints 57<br />
4.6Experimentingwitharchitectureconstraints Intheprevioustestsweonlyusedfullyconnectednets.Nowwewilllook constraint,becauseitcouldalsomakethelearningmoredicultanddecreasethegeneralizationperformance.Thisoccurswhentheweightsharing<br />
onlyhelpthetrainingandgeneralizationwhentheweightsharingisalogical consequence<strong>of</strong>theproblem.<br />
atlocallyconnectednetsandotherconnectionconstraints.Anattractive methodtoreducethenumber<strong>of</strong>weightsisbyforcingcertainweightsto isforcingthenettondasolutionthatdoesnotttheproblem.Itwill beequal.Ofcoursewehavetobeverycarefulwhenweusethistype<strong>of</strong><br />
sucientnumber<strong>of</strong>(dierent)examplesisgiven.Wetraindierentfully 4.6.1Experimentingwithfullyconnectednets Aswehaveseen,thefullyconnectednetscanbetrainedquitewellwhena<br />
feasibletotrainanetwiththatmanyweightsonalargetrainingset. Wendthattwohiddenlayersimprovethetrainingresults,butitisnot inTest1andTest3.Theresults<strong>of</strong>thesetestsaregiveninTable15. connectednetswithuniformsampledinputsandoneortwohiddenlayers<br />
4.6.2Experimentingwithlocallyconnectednets Theadvantage<strong>of</strong>locallyconnectednetsoverfullyconnectednetsisthatthey needmuchlessweightsandtheneuronsinthehiddenlayercanspecializeon theirpart<strong>of</strong>theinput.Thesizeandoverlap<strong>of</strong>thereceptiveeldsarethings<br />
netfromTest16tothefullyconnectednetfromTest3,wehaveobtained tobedetermined.WehavetriedtwolocallyconnectednetsinTest15and<br />
aweightreduction<strong>of</strong>63%.Althoughthetrainingerrorisslightlyworse, Test16.TheresultsaregiveninTable16. number<strong>of</strong>hiddennodesonthesecondhiddenlayer.Whenwecomparethe Theresultsarebetterwhenweusesmallreceptiveeldsandasucient<br />
locallyconnectednetsaremoreattractive(inthisproblem)thanthefully thegeneralization,whichismoreimportant,isalotbetter.Wendthat connectednets.<br />
inSection2.3.3,inthefullyconnectednet(Test17andTable17),thelocallyconnectednet(Test18andTable18)andthewaveletnets(Test19<br />
Inthefollowingtestswehavebuildthesymmetryconstraints,asdescribed 4.6.3Usingsymmetryconstraints<br />
andTable19).
58 4.FORWARDMODELLINGWITHMUDINVASION<br />
Fullyconnected architecture number<strong>of</strong>weights number<strong>of</strong>epochs Table15:Comparingdierentfullyconnectednets.<br />
trainingerror Test1 3128.10.13128.10.10.1 3861 3060 Test3<br />
generalizationerror70.0% 3.1% 3971 6429 1.1% 65.2%<br />
Locallyconnected connections architecture number<strong>of</strong>weights Table16:Comparingdierentlocallyconnectednets.<br />
receptiveeldsize Test15 LocallyconnectedLocallyconnected 3128.12.5.1 1523 3128.19.15.1 Test16<br />
receptiveeldoverlap80% number<strong>of</strong>epochs 7.8feet 1475<br />
trainingerror 2679 3.8feet<br />
generalizationerror 7.5% 61.7% 48.9% 70% 5267 2.1%<br />
Table17:Comparingfullyconnectednetswithoutandwithsymmetryconstraints.Symmetry1<br />
architecture number<strong>of</strong>weights symmetryconstraintsNo 3128.10.13128.10.1 3861 Test1 1941 Yes Test17<br />
number<strong>of</strong>epochs trainingerror generalizationerror 3060 70.0% 3.1% 655 6.6% 37.7%
output node<br />
second hidden layer<br />
<strong>of</strong> 15 nodes<br />
total<br />
number <strong>of</strong><br />
6 feature<br />
maps<br />
(copies)<br />
feature map<br />
(shared weights within map)<br />
4.6Experimentingwitharchitectureconstraints 59<br />
performance.Thisidea<strong>of</strong>symmetryconstraintsisworkedoutfurtherin thetrainingmoredicultforthenet,butitimprovesthegeneralization theconvolutional-regressionnets. Inallteststhenumber<strong>of</strong>weightsisreducedapproximately50%.Thismakes<br />
4.6.4Experimentingwithconvolutionalregressionnets<br />
fromthersthiddenlayeranddeterminesthelocation<strong>of</strong>thefeatureinthe Intheconvolutional-regressionnetsweconstrainallreceptiveeldstoshare input. racyinspatialresolution.Thesecondhiddenlayercombinestheactivations thesameweights.Thesizeandtheoverlap<strong>of</strong>theeldsdeterminetheaccu-<br />
Rt<br />
denlayerareconnectedtoallthevariablesintheinputlayer. Figure36:Convolutional-regressionnet.Thefeaturemapsinthersthid-<br />
Rxo<br />
dxo<br />
Thenumber<strong>of</strong>featuremapsdeterminehowmanyfeaturescanbedetected.<br />
window size<br />
dependuponthenumber<strong>of</strong>featuremapsused(for15hiddennodesin Weshouldnotuseto<strong>of</strong>ewfeaturemaps,otherwisethenethasdicultyin learningtheproblemandingeneralization.Thenumber<strong>of</strong>weights,however,<br />
29.8 feet<br />
receptive field
60 4.FORWARDMODELLINGWITHMUDINVASION<br />
constraints. Table18:Comparinglocallyconnectednetswithoutandwithsymmetry Symmetry2 connections architecture number<strong>of</strong>weights receptiveeldsize Test16 LocallyconnectedLocallyconnected 3128.19.15.1 1475 3128.19.15.1 Test18<br />
receptiveeldoverlap70% symmetryconstraintsNo 3.8feet 905<br />
number<strong>of</strong>epochs Yes 3.8feet<br />
trainingerror 5267 70%<br />
generalizationerror 2.1% 48.9% 37.8% 5536 2.3%<br />
Table19:Comparing\wavelet"netwithoutandwithsymmetryconstraints. Symmetry3 architecture number<strong>of</strong>weights remarks Test9 368.15.1 3091 Test19<br />
symmetryconstraintsNo WaveletcoecientsWaveletcoecients Reduction20,10 Reduction20,10 Yes 367.15.1 1561<br />
number<strong>of</strong>epochs trainingerror generalizationerror 4303 0.8% 51.5% 46.6% 3137 3.6%
4.6Experimentingwitharchitectureconstraints 61<br />
second hidden layer<br />
<strong>of</strong> 15 nodes<br />
total<br />
number <strong>of</strong><br />
9 feature<br />
maps<br />
copies<br />
(3 feature maps)<br />
feature map<br />
(shared weights within map)<br />
Figure37:Convolutional-regressionnet.Thersthiddenlayerconsists<strong>of</strong><br />
Rt<br />
Rxo<br />
dxo<br />
weightsbyk15).Atrade-oshouldbemadebetweenthesizeandoverlap thesecondhiddenlayeronefeaturemap<strong>of</strong>knodesincreasesthenumber<strong>of</strong> toone<strong>of</strong>thevariablesintheinputlayer. threesets<strong>of</strong>featuremaps.Eachsetconsists<strong>of</strong>threemapsandisconnected<br />
window size<br />
<strong>of</strong>theelds,thenumber<strong>of</strong>featuremapstouseandtheresultingnumber<br />
29.8 feet<br />
receptive field<br />
ontheotherhand,thersthiddenlayerwassplitintothreevariablemaps, toallthevariables<strong>of</strong>theinputlayerasshowninFigure36.InTest23, itscorrespondingvariableintheinputlayer.ThisisshowninFigure37. eachcontainingthreefeaturemaps.Eachvariablemapwasconnectedto <strong>of</strong>weightsinthenet.InTest20,21and22thehiddenlayerwasconnected<br />
TheresultsforthesetestsaregiveninTable20andTable21.Forallthese testset.Thegeneralizationperformance<strong>of</strong>thesenetsissignicantlybetter tetsthenetsarelocallyconnectedandthewindowsizeis29.8feet.The convolutional-regressionnetsperformverywellonboththetrainingandthe<br />
generalizationisapproximately2.5timesbetter. thanforallthenetswehavetriedbefore.Whenwecomparetheresults nectednetwithtwolayers),wehaveaweightreduction<strong>of</strong>29%andthe fromTest21withtheresultsfromTest3forexample(Test3isafullycon-
62 4.FORWARDMODELLINGWITHMUDINVASION<br />
thebedboundaries.Itsmoothstheinputsignalandtheactivationsare InFigure39theactivations<strong>of</strong>thersthiddenlayeratacertaindepthare Fromtheseactivationswecanseethersthiddenlayerisactuallydetecting plotted.TheseactivationscomefromTest21. Wehaveinvestigatedthefeaturesthataredetectedbythersthiddenlayer.<br />
constantforparts<strong>of</strong>theinputsignalthatareconstant.
4.6Experimentingwitharchitectureconstraints 63<br />
Convolutional1 architecture number<strong>of</strong>weights samplingperiod Table20:Resultsforconvolutional-regressionnets(1).<br />
receptiveeldsize Test20 360:69:5:13150:627:15:1 503 0.5feet Test21<br />
receptiveeldoverlap75% 5.5feet 2827<br />
gure 0.2feet<br />
number<strong>of</strong>epochs Figure36 3.8feet<br />
trainingerror 3316 75%<br />
generalizationerror 3.9% 34.7% Figure36 2865 0.6% 25.5%<br />
Convolutional2 architecture number<strong>of</strong>weights samplingperiod Table21:Resultsforconvolutional-regressionnets(2).<br />
receptiveeldsize Test22 3100:616:15:13150:927:15:1 1927 0.3feet Test23<br />
receptiveeldoverlap80% 7.2feet 0.2feet 3.8feet 3865<br />
gure number<strong>of</strong>epochs trainingerror Figure36 4081 Figure37 75%<br />
generalizationerror 1.6% 26.3% 0.5% 28.6% 3689
64 4.FORWARDMODELLINGWITHMUDINVASION<br />
Ohm meter<br />
70<br />
Rt<br />
60<br />
50<br />
40<br />
30<br />
Figure38:Trueresistivityprole<strong>of</strong>modelAatdepth615feet.<br />
20<br />
10<br />
0<br />
600 605 610 620 625 630<br />
depth<br />
activation<br />
1<br />
0.8<br />
0.6<br />
Feature map 1<br />
Feature map 2<br />
Feature map 3<br />
Feature map 4<br />
Feature map 5<br />
Feature map 6<br />
0.4<br />
0.2<br />
0<br />
-0.2<br />
-0.4<br />
-0.6<br />
6featuremaps<strong>of</strong>27nodeseach. TheseactivationsareformodelAatdepth615feet.Thislayerconsists<strong>of</strong> Figure39:Activation<strong>of</strong>rsthiddenlayerperfeaturemapforTest21.<br />
-0.8<br />
-1<br />
5 10 20 25 30 35<br />
node
4.7Intermediateresultsarchitecturedesign 65<br />
Wetrainaconvolutional-regressionnetonalargerset<strong>of</strong>models.The 4.7Intermediateresultsarchitecturedesign previousnetworks(28331,14153and47485patternsrespectively).Thenet trainingset,validationsetandtestsetisequaltotheoneusedforthe thatisusedisshowninFigure36.Theperformanceincomparisonwiththe netismuchbetterthanforthepreviousnets. Asonecansee,thegeneralizationperformance<strong>of</strong>theconvolutional-regression previousresultsisshowninFigure40. Wearegoingtousethistype<strong>of</strong>netfortheapproximation<strong>of</strong>boththeDeepandtheShallow-log.<br />
theconvolutional-regressionnets.WetrainanetontheShallow-logand oneontheDeep-log.Weusetheconvolutional-regressionnetfromTest21. Thenetswiththebestperformance(trainingandgeneralizationerror)are 4.8Summaryandresults<br />
Thisnethasaslidingwindow<strong>of</strong>29.8feet,sampledevery0.2feet(3150 inputs),27receptiveelds<strong>of</strong>3.8feetwith75%overlapand6featuremaps. Weperformthefollowingexperiments: Shallow-log(1):Wetrainaconvolutional-regressionnetonearth<br />
Table22:Averagerelativeerrorandperformanceforearthmodelswith aresampledevery0.2feet.TheresultsaregiveninTable22. modelswithinvasion(modelsA,B,C,D,EandF).Thetargetlogs<br />
invasion(Shallow-log). Trainingset Validationsetrelativeerrorpredictedcorrect averagepercentage<strong>of</strong>setnumber<strong>of</strong><br />
Testset 4.3% 5.1% 6.2% 82%correct 77%correct 88%correct patterns 14153 47485 28331<br />
Shallow-log(2):Afterthistrainingweaddmodelswithoutinvasion andsomemodelswithinvasionandrestartthetraining.Forthis arenowsampledevery1.0feet)and40modelswithoutinvasion(target purposeweusethewholetrainingset(seeTable5)(thetargetlogs logswersampledevery0.4feet).TheresultsareshowninTable23. Allmodelscontaineitherinvasionornot.Therefore,theperformance issplitintoinvasionandnoinvasion.
66 4.FORWARDMODELLINGWITHMUDINVASION<br />
percentage <strong>of</strong> log<br />
that is correct<br />
100<br />
95<br />
90<br />
85<br />
80<br />
75<br />
70<br />
Non-Uniform<br />
Haar<br />
Uniform<br />
Convolutional-regression<br />
65<br />
Figure40:Intermediateresults(2)<strong>of</strong>trainingdierentneuralnetson6<br />
60<br />
models.<br />
55<br />
50<br />
Table23:Averagerelativeerrorandperformanceforearthmodelswithand<br />
45<br />
withoutinvasion(Shallow-log).<br />
training validation testing<br />
relativeerrorpredictedcorrect averagepercentage<strong>of</strong>setnumber<strong>of</strong> 5.3% 76%correct patterns<br />
noinvasionTrainingset Validationset 5.5% 75%correct 61688<br />
Testset 5.6% 77%correct 74%correct 14153 14128<br />
10040
4.8Summaryandresults 67<br />
Deep-log:Wetrainaconvolutional-regressionnetonbothmodels<br />
Table24:Averagerelativeerrorandperformanceforearthmodelswithand withandwithoutinvasion.Weusethesametrainingsetasforthe<br />
withoutinvasion(Deep-log). Shallow-log.TheresultsaregiveninTable24.<br />
relativeerrorpredictedcorrect averagepercentage<strong>of</strong>setnumber<strong>of</strong> 8.4% 52%correct patterns<br />
noinvasionTrainingset Validationset 8.8% 9.0% 51%correct 50%correct 61688<br />
Testset 7.0% 6.9% 56%correct 58%correct 14153 14128<br />
scratch.ThenetthatwastrainedontheShallow-logwasoptimalformodels Thelasttwonetsarenotoptimalinasensethattheyarenottrainedfrom10040<br />
invasion.ThenetthatwastrainedontheDeep-logusedtheoptimalset<strong>of</strong> withinvasion.Thenweaddedmodelswithoutinvasionandnewmodelswith weightsfromtherstnetthatwastrainedontheShallow-log(thiswas weights)anddirectly<strong>using</strong>alargetrainingset. resultscanbeachievedbyretrainingthenets(randominitialization<strong>of</strong>the andcontinueditstrainingonnewmodelswithandwithoutinvasion.Better donebecausetheShallow-logandDeep-loglookverysimilarinpractice)
68 5.THENEURALNETWORKASFASTFORWARDMODEL<br />
5Theneuralnetworkasfastforwardmodel wardmodels.Theaccuracyisnotyetsucient,butcanbeimprovedby Inthissectionwepresenttheneuralnetworksthatcanbeusedasfastfor-<br />
trainingonamorerepresentativetrainingset.Foreachnetthe\inputdomain"isgiven.Thisisadescription<strong>of</strong>themodelsthenetistrainedon.<br />
Forearthmodelssimilartothesetrainingmodels,thenetgivesanapproximationwithanaccuracythatisalsogiveninthisdescription.Whenan<br />
<strong>of</strong>(apart<strong>of</strong>)theearthmodel(onlyformodelswithinvasion),theapproximation<strong>of</strong>thelogbytheneuralnetandacumulativeerrorplot.Inthis<br />
cumulativeerrorplotthepercentage<strong>of</strong>thelogthathasarelativeerrorbelowR%(forR=0:::25)isgiven.Inallplotstheinvasionradiusisgivenin<br />
earthmodelliesoutsidethisdomain,onecannotexpectthenettoperform optimal. Foreachnetthebestandworsttestresultisgiven.Theseresultsconsist<br />
Adescription<strong>of</strong>thisnetcanbefoundinChapter3.Thenetisfullyconnected.TheinputdomainforthenetthatistrainedontheDeep-andthe<br />
net toolresponse sizeslidingwindow sizetrainingset rangeRt rangebedsize Deep-logandShallow-log<br />
5.1Applicationtoearthmodelswithoutinvasion 0.1inchandtheresistivitiesinm.<br />
Shallow-log,herecalledNoInvasion,isgiveninthefollowingtable:<br />
speed 1...70m 1...5feet 11797examples 400pnt/sec 15feet<br />
InFigures41and42theworstandbesttestresultsareshownforthe accuracyDeep-log accuracyShallow-log 2.2%<br />
Deep-logapproximation<strong>of</strong>thenetNoInvasion.Fromthecumulativeerror 6.0%<br />
plotsfortheseapproximationsonecanseethatthedeviationinperformance 5%,butinthebestcasethisis100%. isquitelarge.Intheworstcaseonly8%<strong>of</strong>theloghasarelativeerrorbelow InFigures43and44theworstandbesttestresultsareshownforthe manceismuchlessthaninthecase<strong>of</strong>theDeep-log.Intheworstcase75% Shallow-logapproximation<strong>of</strong>thenetNoInvasion.Thedeviationinperfor-<br />
<strong>of</strong>theloghasarelativeerrorbelow5%andinthebestcasethisis100%.
5.1Applicationtoearthmodelswithoutinvasion 69<br />
100<br />
True resistivity Rt<br />
Deep-log<br />
Approximation by neural net<br />
10<br />
1<br />
0 5 10 15 20 25 30 35 40 45 50<br />
Percentage <strong>of</strong> log<br />
with relative error<br />
below R %<br />
100<br />
90<br />
80<br />
70<br />
60<br />
50<br />
40<br />
30<br />
Figure41:Worstcaseneuralnetapproximation<strong>of</strong>Deep-log.Averagerelativeerroris14.2%.<br />
20<br />
10<br />
0<br />
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />
Relative error R
70 5.THENEURALNETWORKASFASTFORWARDMODEL<br />
100<br />
True resistivity Rt<br />
Deep-log<br />
Approximation by neural net<br />
10<br />
1<br />
0 5 10 15 20 25 30 35 40 45 50<br />
Percentage <strong>of</strong> log<br />
with relative error<br />
below R %<br />
100<br />
90<br />
80<br />
70<br />
60<br />
50<br />
40<br />
30<br />
Figure42:Bestcaseneuralnetapproximation<strong>of</strong>Deep-log.Averagerelative erroris2.6%.<br />
20<br />
10<br />
0<br />
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />
Relative error R
5.1Applicationtoearthmodelswithoutinvasion 71<br />
100<br />
True resistivity Rt<br />
Shallow-log<br />
Approximation by neural net<br />
10<br />
1<br />
0 5 10 15 20 25 30 35 40 45 50<br />
Percentage <strong>of</strong> log<br />
with relative error<br />
below R %<br />
100<br />
90<br />
80<br />
70<br />
60<br />
50<br />
40<br />
30<br />
Figure43:Worstcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris8.5%.<br />
20<br />
10<br />
0<br />
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />
Relative error R
72 5.THENEURALNETWORKASFASTFORWARDMODEL<br />
100<br />
True resistivity Rt<br />
Shallow-log<br />
Approximation by neural net<br />
10<br />
1<br />
0 5 10 15 20 25 30 35 40 45 50<br />
Percentage <strong>of</strong> log<br />
with relative error<br />
below R %<br />
100<br />
90<br />
80<br />
70<br />
60<br />
50<br />
40<br />
30<br />
Figure44:Bestcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris1.2%.<br />
20<br />
10<br />
0<br />
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />
Relative error R
5.2Applicationtoearthmodelswithinvasion 73<br />
Shallow-log.Itistrainedonearthmodelsasdescribedinthefollowingtable: regressionnet.Thefollowingnetonlyproducesanapproximation<strong>of</strong>the Adescription<strong>of</strong>thisnetcanbefoundinChapter4.Thisisaconvolutional-<br />
5.2Applicationtoearthmodelswithinvasion<br />
net toolresponse sizeslidingwindow sizetrainingset rangeRt Invasion<br />
rangeRxo rangedxo 0.5...2.5m 1...71m 8...50inch 28331examples 30feet<br />
rangebedsize speed 1...20feet<br />
InFigure45two<strong>of</strong>thetestmodelsareshown.Themodelatthetopis accuracyShallow-log 6.2% 90pnt/sec<br />
Inthecumulativeerrorplot<strong>of</strong>therstapproximationisshownthat70%<strong>of</strong> thelogisapproximatedwitharelativeerrorbelow5%.Inthecumulative proximation<strong>of</strong>thenetInvasionareshown. dicultfortheneuralnetandthemodelatthebottomrelativelyeasy.In Figures46and47theworstandbesttestresultsfortheShallow-logap-<br />
errorplot<strong>of</strong>thesecondapproximationonecanseethatinthebestcase 83%<strong>of</strong>thelogisapproximatedwitharelativeerrorbelow5%. Thisnetworkisoptimalfortheearthmodelsitwastrainedon.<br />
on,areacombination<strong>of</strong>thepreviousmodelswithoutandwithinvasion.A remarkshouldbemadethatthevalueRxowastakentobe1.1mwhen Thedomain<strong>of</strong>earthmodelstheconvolutional-regressionnetFMistrained 5.3Applicationtorealisticearthmodel<br />
fromarealoilwell.Thismodelismorerealistic.Itdoesnotlieintheinput dxowas4.25inch.InFigure48aformationmodelisshownthatcomes domain,becauseitcontainsbothpartswithandwithoutinvasion.Inour trainingseteachmodelonlycontainedeitherinvasionornot.Thebedsare relativelysmall.TheDeep-logandShallow-loglookverysimilarforthis donotexpectthenettobeoptimalforthismodel. thismodellooksquitedierentfromthemodelsinthetrainingsetandwe model,thisisalsodierentfromthemodelsthenetistrainedon.Allinall
74 5.THENEURALNETWORKASFASTFORWARDMODEL<br />
Adescription<strong>of</strong>theearthmodelsthenetwastrainedonisgiveninthe followingtable: net toolresponse sizeslidingwindow sizetrainingset rangeRt Deep-logandShallow-log 61688examples FM<br />
rangeRxo rangedxo 4.25,8...50inch 0.5...2.5m 1...71m 30feet<br />
rangebedsize speed accuracyShallow-log accuracyDeep-log 1...20feet 8.0% 9.0% 85pnt/sec<br />
imation<strong>of</strong>theShallow-logisshowninFigure50.Aswecanseefromthe Theapproximation<strong>of</strong>theDeep-logisshowninFigure49andtheapprox-<br />
cumulativeerrorplots,therelativeerrorbetweentheneuralnetresponse andtheDeep-logisbelow5%for67%<strong>of</strong>thelog.For88%<strong>of</strong>thelogthe relativeerrorliesbelow10%. TheShallow-logapproximationismuchbetter.Here77%<strong>of</strong>theloghasa relativeerrorbelow5%.
5.3Applicationtorealisticearthmodel 75<br />
100<br />
True resistivity Rt<br />
Invasion resistivity Rxo<br />
Invasion radius dxo<br />
10<br />
1<br />
200 210 220 230 240 250 260 270 280 290 300<br />
100<br />
True resistivity Rt<br />
Invasion resistivity Rxo<br />
Invasion radius dxo<br />
10<br />
Figure45:Examples<strong>of</strong>earthmodelsusedforthenetInvasion.<br />
1<br />
200 210 220 230 240 250 260 270 280 290 300
76 5.THENEURALNETWORKASFASTFORWARDMODEL<br />
Shallow-log<br />
Approximation by neural net<br />
100<br />
10<br />
1<br />
200 210 220 230 240 250 260 270 280 290 300<br />
Percentage <strong>of</strong> log<br />
with relative error<br />
below R %<br />
100<br />
90<br />
80<br />
70<br />
60<br />
50<br />
40<br />
30<br />
Figure46:Worstcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris7.5%.<br />
20<br />
10<br />
0<br />
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />
Relative error R
5.3Applicationtorealisticearthmodel 77<br />
Shallow-log<br />
Approximation by neural net<br />
100<br />
10<br />
1<br />
200 210 220 230 240 250 260 270 280 290 300<br />
Percentage <strong>of</strong> log<br />
with relative error<br />
below R %<br />
100<br />
90<br />
80<br />
70<br />
60<br />
50<br />
40<br />
30<br />
Figure47:Bestcaseneuralnetapproximation<strong>of</strong>Shallow-log.Average relativeerroris4.5%.<br />
20<br />
10<br />
0<br />
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />
Relative error R
78 5.THENEURALNETWORKASFASTFORWARDMODEL<br />
True resistivity Rt<br />
Invasion resistivity Rxo<br />
Invasion radius dxo<br />
100<br />
Figure48:A(realistic)earthmodel.<br />
10<br />
1<br />
5400 5410 5420 5430 5440 5450 5460 5470 5480 5490 5500
5.3Applicationtorealisticearthmodel 79<br />
Deep-log<br />
Approximation by neural net<br />
100<br />
10<br />
1<br />
5400 5410 5420 5430 5440 5450 5460 5470 5480 5490 5500<br />
Percentage <strong>of</strong> log<br />
with relative error<br />
below R %<br />
100<br />
90<br />
80<br />
70<br />
60<br />
50<br />
40<br />
30<br />
Figure49:<strong>Neural</strong>netapproximation<strong>of</strong>Deep-log.Averagerelativeerroris 7.6%.<br />
20<br />
10<br />
0<br />
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />
Relative error R
80 5.THENEURALNETWORKASFASTFORWARDMODEL<br />
Shallow-log<br />
Approximation by neural net<br />
100<br />
10<br />
1<br />
5400 5410 5420 5430 5440 5450 5460 5470 5480 5490 5500<br />
Percentage <strong>of</strong> log<br />
with relative error<br />
below R %<br />
100<br />
90<br />
80<br />
70<br />
60<br />
50<br />
40<br />
30<br />
Figure50:<strong>Neural</strong>netapproximation<strong>of</strong>Shallow-log.Averagerelativeerror is8.3%.<br />
20<br />
10<br />
0<br />
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25<br />
Relative error R
6Conclusions Inthissectionwepresenttheconclusionsfollowingfromtheexperimentswe haveperformedandthemethodswehaveinvestigated.Theconclusionsare splitintothreeparts 1.Analconclusionaboutthegoal<strong>of</strong>thisproject:investigatingwhether 2.Conclusionsabouttheusedmethods.Thisinvolvestheinputrepresentation,thetechniquesthatareusedtoreducethenumber<strong>of</strong>inputs<br />
itisfeasibletouseaneuralnetworkintheforwardmodellingprocess. andthearchitecturalconstraints. 3.Conclusionsabouttheuse<strong>of</strong>theneuralnetworkintrainingandtesting.Theseconclusionsareusefulinfurtherinvestigationandinthe<br />
Intherstpart<strong>of</strong>theprojectweonlyusedearthmodelswithoutinvasion 6.1<strong>Neural</strong>networkasfastforwardmodel? use<strong>of</strong>thetrainednetworkinotherapplications.<br />
types<strong>of</strong>earthmodelsneuralnetsweretrainedwiththefollowingresults: andinthesecondpartweonlyusedearthmodelswithinvasion.Forboth Noinvasion:Weusea\standard"fullyconnectednetwithonehiddenlayercontaining15nodes,aslidingwindow<strong>of</strong>15feetandasamplingperiod<strong>of</strong>0.2feet(75inputs).Thenetworkhasmoretrouble<br />
inlearningtheDeep-logthantheShallow-log.Thisiscausedbythe shoulderbedeect,whichismorepronouncedintheDeep-logthanin theShallow-log.Theperformanceisgiveninthefollowingtable TargetlogDataset Shallow-logTrainingsetwithrelativeerrorrelativeerror<br />
percentage<strong>of</strong>log below5%. 98% average<br />
Deep-log Testset 96% 62% 61% 1.7% 2.2% 5.1% 6.0%<br />
81
82 6.CONCLUSIONS<br />
Invasion:Wetrainedoneconvolutional-regressionnetontheShallowlog.Thenetusesaslidingwindow<strong>of</strong>29.8feet,sampledevery0.2feet,<br />
Theperformanceisgiveninthefollowingtable TargetlogDataset Shallow-logTrainingset withrelativeerrorrelativeerror percentage<strong>of</strong>log below5%. 88% average 27receptiveelds<strong>of</strong>3.8feetwith75%overlapand6featuremaps.<br />
Validationset 82% 4.3%<br />
Mixedinvasionandnoinvasion:Weusedthesameconvolutionalregressionnetasintheprevioustestswithatrainingsetconsisting<strong>of</strong><br />
6.2% Testset 77% 5.1%<br />
wetrainedthenetonboththeDeep-andtheShallow-log.Thesenets earthmodelswithandwithoutinvasion(butnotmixed).Thistime<br />
modelswithandwithoutinvasion Inthefollowingtabletheperformance<strong>of</strong>thenetsaregivenforthe arenotoptimal,becauserstthenetwasminimalizedonmodelswith invasionandthenweaddedmoremodelswithandwithoutinvasion. Targetlog Shallow-loginvasion Dataset relativeerror below5%. percentageaverage <strong>of</strong>logwithrelative 76% error<br />
noinvasionTrainingset Validationset 75% 77% 5.5%<br />
5.3%<br />
Deep-log 74% 52% 5.6%<br />
noinvasionTrainingset Validationset Testset 51% 50% 58% 56% 8.8% 9.0% 6.9% 8.4% 7.0%
6.2Methods 83<br />
Realloggingdata:Realearthmodelscontainbothlayerswithand earthmodel.Theperformance<strong>of</strong>thesenetsontherealloggingdata isTargetlog themixedinvasionandnoinvasionpartweretestedonarealistic withoutinvasion.Thetrainedconvolutional-regressionnetworksfrom<br />
Shallow-logwithrelativeerrorrelativeerror<br />
percentage<strong>of</strong>log below5%. 77% average<br />
Deep-log 67% 8.3%<br />
this(partlywithandpartlywithoutinvasion)andbychoosingthe Thenetworkperformancecanbeimprovedby<strong>using</strong>moremodelslike 7.6%<br />
Approximationtime:Thegoal<strong>of</strong>theprojectistocreateafaster thantheforwardmodelthatispresentlyusedatKSEPL(bothtimes forwardmodel.Theneuralnetworkisapproximately100timesfaster invasionradiusbetween4and25inchinstead<strong>of</strong>8and50inch.<br />
Althoughtheaccuracy<strong>of</strong>theapproximationstillneedssomeimprovement, thenetworkisalotfasterthantheforwardmodelthatisusednow.We measuredonanIBMR6000workstation).Thiscouldstillbeimproved<br />
concludethatitisfeasibletouseaneuralnetworkintheforwardmodelling byoptimization<strong>of</strong>theneuralnetcalculations.<br />
process.Evenwithlessaccuracytheneuralnetworkcouldbeusedinthe rstiterations<strong>of</strong>theforwardmodellingprocess.Inthatwayafairlygood initialguesscanbemadeveryquickly.Thenonecanusethemoreaccurate forwardmodel. 6.2Methods<br />
whereweusedearthmodelswithinvasion. Intherstpart<strong>of</strong>theprojectweonlyusedfullyconnectednetswithvarying window.Moreinterestingresultsarefoundinthesecondpart<strong>of</strong>theproject, number<strong>of</strong>layers,hiddennodesandvariationsinthesize<strong>of</strong>thesliding<br />
6.2.1Inputrepresentation Weexperimentedwithtwoinputrepresentation.Therst,thediscretized andhasbeenusedinfurthertests: slidingwindowapproach,gavethebestresults(trainingandgeneralization) Discretizedslidingwindow:Weusedauniformsampledsliding windowasinputtotheneuralnet.Thenumber<strong>of</strong>inputscomingfrom
84 6.CONCLUSIONS<br />
Attributes:Againweusedaxedsizeslidingwindow,butnowwe well. trainingresultwasgood,butthenetworkwasnotabletogeneralize thiswindowcanbequitehigh,especiallyinthecasewithinvasion.The<br />
describethebedsthatoccurinthiswindow.Eachbedwasdescribed byanumber<strong>of</strong>attributes.Theproblem,however,wasnotdescribed<br />
6.2.2Inputreduction task. wellbytheseattributes.Tondappropriateattributesisadicult<br />
Allpreprocessingmethodsweusedweresuccessful.Weachievedahigh inputreductionwithoutloss<strong>of</strong>performance(measuredintrainingandgeneralizationresults):<br />
Samplingmethod:Theslidingwindowcanbesampleduniformand plingperiods.Acoarsesamplingperiodresultsinlessinputsandthe non-uniform.Intherstcaseweexperimentedwithdierentsam-<br />
fromtheedges.Non-uniformsamplingresultsinlessinputsandthe receivesmost<strong>of</strong>itsinformationfromthecenter<strong>of</strong>thewindowandless ariesaredescribedlessaccurately.Thenon-uniformsamplingmethod workedverywell.Thismethodisbasedonthetoolphysics.Thetool trainingandgeneralizationresultswerecomparable.Thebedbound-<br />
Principalcomponents:Withtheprojection<strong>of</strong>theN-dimensional generalizationperformance<strong>of</strong>thenetwasbetterthanforthenetthat usedauniform-sampledslidingwindowasinput.Hereweonlyloose accuracyattheedges<strong>of</strong>thewindow.<br />
tion<strong>of</strong>theinputwaslost(measuredintherelativedistancebetween originaldiscretizedinput,althoughapproximately7%<strong>of</strong>theinforma-<br />
resultswerecomparableandthenetworkgeneralizedbetterthanthe components,aninputreduction<strong>of</strong>75%wasachieved.Thetraining inputvectortoaM-dimensionalvector,describedbyMprincipal<br />
Haartransform:Theadvantage<strong>of</strong>transformation<strong>of</strong>theoriginal cientscanberemoved.Areduction<strong>of</strong>73%<strong>of</strong>thenumber<strong>of</strong>inputs inputvaluestothewaveletcoecientsisthatanumber<strong>of</strong>thecoe-<br />
theoriginalandprojectedinput).<br />
wasachievedwithgoodtrainingandgeneralizationresults.Byremovingcoecientsattheedges<strong>of</strong>thewindow,weloosesomeaccuracy.<br />
lessaccurately. Thebedboundariesoutsidethecenter<strong>of</strong>thewindowaredescribed
6.3Application<strong>of</strong>theconvolutional-regressionnet 85<br />
6.2.3Architecturedesign Fullyconnectednets:Inthecasewithoutinvasionafullyconnected<br />
resultinginalongtrainingtimeandstorageandmemoryproblems. generalizewell.Thenumber<strong>of</strong>connectionsinthisnetisveryhigh, threeparameterstodescribethebed.Thistype<strong>of</strong>networkdoesnot netcanbeused.Thegeneralizationperformanceandthetraining<br />
Preprocessingmethodscanbeusedtoreducethenumber<strong>of</strong>inputs. resultsarecomparable.Whenthebedscontaininvasion,weneed<br />
Locallyconnectednets:Aneuralthatislocallyconnectedhas fullyconnectednets.Thesize<strong>of</strong>thereceptiveeldsandtheoverlap largeoverlap.Inthiswayitiseasierforthenettodeterminethe nets.Thetrainingtimeneededforthesenetsisshorterthanforthe arediculttodetermine.Wechoose,however,forsmalleldswith muchlessweightsandbettergeneralizationthanthefullyconnected<br />
Symmetryconstraints:Aninputsignalanditsmirrorimagegive preciselocation<strong>of</strong>afeaturethatoccursinone<strong>of</strong>theelds.Thisnet needsatleasttwohiddenlayers:onefordetectingthelocalfeatures andoneforcombiningthefoundfeatures. thesametoolresponseandshouldthereforealsogivethesamenetworkresponse.Thisrequirementleadstocertainweightconstraintduced,becausecertainweightsare\shared"(equal).Thereduction<br />
inthefully,locallyandwaveletnets.Thenumber<strong>of</strong>weightsisre-<br />
Convolutional-regressionnets:Intheconvolutional-regressionnet inthenumber<strong>of</strong>weightsmakesthetrainingmoredicult,butitimprovesthegeneralizationperformanceincomparisonwiththesame<br />
allreceptiveeldssharethesameweights.Onegroup<strong>of</strong>hiddennodes withsharedweightsiscalledafeaturemap.Thereduction<strong>of</strong>freedom netswithouttheseconstraints.<br />
set<strong>of</strong>sharedweights,thatishaslearneditself.Thegeneralization performance<strong>of</strong>thistype<strong>of</strong>netisbetterthanforalltheothernetswe iscompensatedbyanincreaseinthenumber<strong>of</strong>featuremaps.This netperformsaconvolutionontheinputwithaconvolutionkernel,the<br />
Whentheconvolutional-regressionnetisgoingtobeusedinotherapplications,thefollowingaspectsareimportant:<br />
Size<strong>of</strong>theslidingwindow:Thesize<strong>of</strong>theslidingwindowdepends ontheapplication.Oneshouldhaveanidea<strong>of</strong>howmuch<strong>of</strong>theinput<br />
6.3Application<strong>of</strong>theconvolutional-regressionnet havetried.<br />
logisresponsiblefortheoutput.
86 6.CONCLUSIONS<br />
Samplingperiod<strong>of</strong>slidingwindow:Thesamplingperiod<strong>of</strong>the<br />
Receptiveelds:Thesizeandoverlap<strong>of</strong>thereceptiveeldsdeterminetheaccuracy<strong>of</strong>thelocation<strong>of</strong>thefeaturesintheinput.Use<br />
goodindicationistousethesamesamplingperiodasforthetarget slidingwindowdetermineshowaccuratetheinputisdescribed.A<br />
smalleldswithhighoverlap.Thenumber<strong>of</strong>weights,however,dependonthenumber<strong>of</strong>receptiveelds.Choosethesize<strong>of</strong>theelds<br />
andtheoverlapsothatthenumber<strong>of</strong>weightsisnottoolarge. thetargetlogs.Donottakeatoodensesampling<strong>of</strong>thetargetlog, otherwisemostexampleslooktoomuchalike.Itisbettertotakea<br />
log.<br />
Representativetrainingset:Thetrainingsetisconstructedfrom<br />
Sizetrainingset:Thenumber<strong>of</strong>trainingexamplesthatareneeded (very)coarsesamplingperiodandusealargenumber<strong>of</strong>dierentlogs. examples.Createmodelsthatarelikelytobefoundinreality. Inthiswaythetrainingsetconsists<strong>of</strong>alargenumber<strong>of</strong>verydierent inordertogetgoodgeneralizationperformancedependsuponthe<br />
Minimalizationnetworkerror:Theerrorthatthenetworkisminimalizingcanbeadaptedtotherequirements<strong>of</strong>theapproximation.<br />
thenumber<strong>of</strong>weightsasexamples. ahighnumber<strong>of</strong>dierentexamples,oneneedsapproximately10times representativeness<strong>of</strong>thetrainingset.Whenthetrainingsetcontains<br />
Withthecombinedlogarithmicandnormalizationscalingtheneural<br />
morevariablesthanRt,Rxoanddxo.Whenthereareforexampledipping Theconvolutional-regressionnetcaneasilybeadaptedforproblemswith andaistheactualoutput. netisminimalizingtheproportiond=a,wheredisthedesiredoutput<br />
intheformation.Thescaling<strong>of</strong>anewvariableneedsnewinvestigation. Theboreholeradiusandresistivity<strong>of</strong>thedrillinguidcanalsobeadded asvariables,butlikethedipangle,thisshouldonlybedonewhenthereare point).Thisisonlypossiblewhenthereisadierentdipangleforeachbed layers,thedipanglecanbeaddedasfourthvariable(foreachsampling<br />
enoughdierentvalues.
A<strong>Neural</strong>networksimulators (vanCamp1993).Wealsotriedothernetworksimulators,Stuttgart<strong>Neural</strong> InthisprojectweusedtheXerionnetworksimulatorversions3.1and4.0 NetworkSimulator(SNNS)(Zell1993)andAspirin/MIGRAINES(Leighton designonanylevel:layers,nodesandevenconnections.Thisisveryuseful 1992),butwefoundthatXerionprovidesmorefreedominspecifyingarchitectureconstraints.TheXerionsimulatorallowsyoutoalterthenetwork<br />
fortheimplementation<strong>of</strong>theconvolutional-regressionnets,whereweconstraincertainweightstobeequal.<br />
optionfortimedelayedneuralnets(seeSection2.3.5).Alayerisspecied byitsnumber<strong>of</strong>features(orvariables)anditstotaldelaylength(number<strong>of</strong> TheSNNSsimulatorisabeautifulgraphicalsimulatorandhasaspecial<br />
Anditisalsonotpossibletochangetheoverlap<strong>of</strong>thereceptiveelds.We Itis,however,notpossibletoconstrainarbitraryweightstobethesame. eldsarespeciedbytheirsize(delaylength)andalwayshavemaximum overlap(displacement<strong>of</strong>onenode).Allreceptiveeldshavesharedweights. nodesinslidingwindow).Layerscanbefullyorlocallyconnected.Receptive<br />
presenttheinputsasasequencewithinaslidingwindow,butfortheSNNS<br />
periods.Thedelaylengthspecieshowmanyinputsbeforethisinputaect simulatoroneshouldpresenttheinputsatonespecicloggingpoint.The<br />
theoutput.Onecannotspecifyhowmanyinputsafterthisinputaectthe inourcase.Thereareonly4valuesperpattern(Rt,Rxo,dxoandthe advantage<strong>of</strong>thisisthatthelethatcontainsthedataismuchsmallerthan<br />
output.Sotherearethreemainpointswhythissimulatorisnotsuitable forourpurposes: toolresponse)instead<strong>of</strong>3(ws+1)forawindowsizewandasampling<br />
3.Onecanonlyspecifyhowmanyinputsbefore(andnotafter)the 2.Theoverlap<strong>of</strong>thereceptiveeldsisalwaysmaximal. 1.Itisnotpossibletospecifyarbitraryweightconstraints.<br />
Aspirin/MIGRAINESisacompiler.Itcompilesyourcodeintoaworkingneuralnetworksimulator(inC).Inthesourcecodetheneuralnetis<br />
Itisnotpossibletospecifyconvolutional-regressionnetswiththissimulator. currentinputaecttheoutput.<br />
layer,thesize<strong>of</strong>thelayer(innodes)andconnectioninformation(howthe speciedbyitscomponents.Acomponentisdescribedbythename<strong>of</strong>the calledSharedTessellationsandonecanspecifythesize<strong>of</strong>theeldsandthe overlapinthex-andy-direction.Itisnotpossible,however,toconstrain layerisconnectedtootherlayers).Receptiveeldswithsharedweightsare specicweightstobeshared,whichweneededinthesymmetryexperiments.<br />
I
II A.NEURALNETWORKSIMULATORS<br />
Itispossibletocreateconvolutional-regressionnetswiththissimulator. Althoughreceptiveeldsareeasierdenedbytheprevioussimulators,we haveusedtheXerionsimulator.Allhiddenneuronsinonefeaturemap areconnectedbynmlinkstotheircorrespondingreceptiveeld(<strong>of</strong>size nm).Alltheseconnectionshavetobespecied,whichcanmountupto 1620connectionsintheconvolutional-regressionnet.Neuronscanbeconstrainedtohavethesameincominglinks,whichiswhatwedidperfeature<br />
reading<strong>of</strong>thisleandthebuilding<strong>of</strong>thenetisquiteslow,butitoutweighs thelack<strong>of</strong>freedom<strong>of</strong>theothersimulators. weights.OurXeriontopologylescontainedatotal<strong>of</strong>9951lines.The map.Thenthenodesinonefeaturemapareconstrainedtohavethesame
REFERENCES<br />
References Anderson,B.&Barber,T.(1990).<strong>Modelling</strong>electromagnetictoolresponse,<br />
Chemali,R.,Gianzero,S.&Strickland,R.(1983).Theshoulderbedeffectontheduallaterologanditsvariationwiththeresistivity<strong>of</strong>the<br />
Asquith,G.(1982).Basicwellloganalysisforgeologists. OileldReviewpp.22{32.Welllogging.<br />
Chemali,R.,Gianzero,S.&Su,S.(1988).Theduallaterologincommon complexsituations,SPWLA29thannualloggingsymposium. boreholeuid,SPWLA24thannualloggingsymposium.<br />
Gianzero,S.(1977).Characteristicresponses<strong>of</strong>resistivitytoolsinelliptical Gianzero,S.,Lin,Y.&Su,S.(1985).Anewhigh-speedhybridtechnique boreholes,IEEEtransactionsongeoscienceelectronicsGE-15(4):251{ forsimulationandinversion<strong>of</strong>resistivitylogs. 256.<br />
Guyon,I.(1991).<strong>Neural</strong>networksandapplicationstutorial,PhysicsReports Haykin,S.(1994).<strong>Neural</strong><strong>Networks</strong>,AComprehensiveFoundation,MacmillanCollegePublisingCompany,Inc.<br />
(Reviewsection<strong>of</strong>PhysicsLetters)207(3{5):215{259.ReviewSection <strong>of</strong>PhysicalLetters207.<br />
Hertz,J.,Krogh,A.&Palmer,R.(1991).Introductiontothetheory<strong>of</strong> Leighton,R.R.(1992).TheAspirin/MIGRAINES<strong>Neural</strong>NetworkS<strong>of</strong>tware,v6.0edn.<br />
neuralcomputation,Addison-WesleyPublishingCompany.<br />
Strang,G.(1989).Waveletsanddilationequations:Abriefintroduction, Moran,J.(1985).Focusedresistivitylogs,Developmentsingeophysical SIAMReview31(4). explorationmethods6:225{260.<br />
vanCamp,D.(1993).AUsersGuideforTheXerion<strong>Neural</strong>NetworkSimulator,ver3.1&ver4.0edn,University<strong>of</strong>Toronto.<br />
Waibel,A.,Hanazawa,T.,Hinton,G.,Shikano,K.&Lang,K.(1989). Phonerecognition<strong>using</strong>time-delayneuralnetworks,IEEETransactionsonacoustics,speechandsignalprocessing37(3).<br />
Zell,A.(1993).SNNSStuttgart<strong>Neural</strong>NetworkSimulator,v3.1edn,InstituteforParallelandDistributedHighPerformanceSystems.<br />
III