Estimation of projection angle using reference
VIII Determining Projection Angle When Reference Image is Available
This section describes how to determine the projection angle if a reference image is available. When restructuring a 3D model, if a reference model is available, the 3D image can be determined by predicting a projection image based on the reference model and determining a direction. To determine the projection angle with reference, a reference image is needed, which may be simply a sphere or an ellipse. The structure analysis method applied when no reference image is usable is also available separately.
VIII-1. Outline The procedure is described below.
1. Prepare a reference image (3D).
2. Create reference images (2D) from the 3D image as reference projection images. The respective reference projection images have a determined angle.
3. Compare the cut-out 2D projection image with the reference projection images created at step 2, select the most similar reference projection image, and estimate the projection angles of the respective 2D images. On this occasion, rotate and move the projection images to the position and rotational direction which is the most suitable for that reference image, and obtain a set of projection images for the purpose of 3D reconfiguration.
4. Reconfigure a 3D model from the set of projection images using the estimated projection angles.
VIII-2. How to determine a similar image The method utilizing a correlation function is used most frequently. For this function, a little ingenuity is required so that better correlation is obtained by pre-processing this function to reduce influence due to noises. On Eos, the mrcImageAutoRotationCorrelation command is used. This program estimates a reasonable angle using correlation functions (normal one, phase correlation, or phase correlation with intensity weight) as considering in-plane rotation. The mrcImageAutoRotationCorrelation command naturally succeeds in most cases if there is no noise and the reference image is appropriate. Therefore, an appropriate pre-processing is necessary. For example, reduce noises using mrcImageSmoothing or mrcIageLowPassFilter. In addition, the correlation functions themselves must be considered. The phase correlation function with weight (phase correlation method with strong route weight).
VIII-3. Concrete exercise This exercise describes a 3D reconfiguration method utilizing Eos. First, download a file of exercise data, WorkshopData2.tar.gz, and decompress it. The decompression method is as follows:
$ tar xvzf WorkshopData2.tar.gz
By executing this, a directory named EosForData2 is generated. There are some Makefiles in it.
$ cd EosForData2/SingleParticlesWithRef
Execute the above command to change the directory. In this directory, perform steps one by one in the following order. Type like below,
$ make
the execution method is displayed as follows:
Makefile for Single Particle with Reference
1. Prepare files from EM images, which has a suffix of roi
2. Prepare an initial model as initial.3d
3. make First
4. make All If you want to repeat refinement again,
1. make Again
2. make All
--- If you want to create an initail model from PDB ---
$ make InitialModel
--- If you want to create an test from the initial model --- $ make TestData
According to this method, the first-time turn can be calculated by specifying below:
$ make First $ make All
More than one turn is calculated by setting as follows:
$ make Again $ make All
More than one calculation is performed by defining as follows:
$ for i in `seq 1 5`; do make Again; make All; done
This section describes the details below.
VIII-3.(1) Preparation of a reference image Prepare a reference image. In this section, a 3D image created from a PDF file is prepared. Preparation can be executed using the command below.
$ make Initial
To change the PDF file used as a reference image, change the following portion in the Makefile.
REFSOURCE=121p-shift DELTA=2.5 SIZE_X=64 SIZE_Y=64 SIZE_Z=64
REFSOURCE is the name of the PDF file referred to. For this Makefile, it is expected that the origin is located around the center of the molecule. Since information on the PDF file can be obtained by pdbInfo, the image can be moved using pdbMove so that the center of gravity is positioned in the origin. The 121p-shift.pdb file is created by moving an atomic model of H-ras called 121p.pdb using the above procedure. DELTA means the step size of the image. The reference image must be created according to the step size of the RIO image used. Although the step size can be changed afterward, setting it here will make things easy. SIZE_X|Y|Z defines the size of the image. These parameters are also related to the size of the ROI image used. These sizes must correspond to the file padded by mrcImagePad, which is described later. Calculation often starts recently without defining a shape but using a sphere or an ellipsoid. This is to minimize the influence of the reference image. If beginning with a sphere, the reference image can be created using mrcImageSphere. If using mrcImageGaussianSphere, the Gauss sphere of an ellipsoid can be set up. The mrcImageGaussianDisc command can make the intensity of the Gauss sphere of the ellipsoid constant.
VIII-3.(2) Preparation of a ROI file Cut a ROI file out of the image, and prepare it. Use Display2. This time, use the test data for confirming if the program is operating well. Test data can be newly created by executing the following:
$ make TestData
This method is used this time. By specifying like below,
$ make TestDataNoise
test data with noises can be created. By changing the value of the line below,
SNRATIO=3
the noise level can be changed. This is often used for testing. By changing the noise level, it is tested that to what level the algorithm can bear. Data with noises and data without noises are included together in the provided data.
VIII-3.(3) Flow of 3D reconfiguration (processing by make All)
The basic flow of single particle analysis is as follows: Creation of a reference projection 2D image: .ref3d -> .ref2d The speed of calculation the precision of reconfiguration are determined by setting up the reference image. Since the Euler angle is YOYS, the 3D image is to be rotated in order from Y, X, and Z. It is rotated by 30 degrees at a time in a range of 0-359 degrees.
# # Search Area for 3D # ROTMODE=YOYS # Rot1: Rotation around the Y axis is set. ROT1MIN=0 ROT1MAX=359 ROT1D=30 nRot1=12 # Rot2: Rotation around the X axis is set. ROT2MIN=0 ROT2MAX=359 ROT2D=30 nRot2=12 # Rot3: Rotation around the Z axis is set. This rotation is realized using AutoRotationCorrelation, too. Which command should be used is determined depending on the available memory size. ROT3MIN=0 ROT3MAX=0 ROT3D=30 nRot3=1
Pre-processing of the projected image: .roi -> .pad The cut-out *.roi file is pre-processed, and converted into a *.pad file. Windowing and Padding are applied to the image used this time. In addition to this, smoothing, frequency filtering, or masking might be performed. In addition, size-matching with the reference image is also performed in this phase. Since a PDF file is the reference image this time, specify the same value between
PAD_W=64 PAD_H=64
and
SIZE_X=64 SIZE_Y=64.
It is very often to conduct 3D reconfiguration with an average image after cluster analysis is performed. In such a case, if using this Makefile, it is also possible to prepare that file as converting .avg into .pad instead of converting .roi into .pad.
Comparison of the projection image with a reference projection 2D image: .pad.corinfo The 2D image projected from the reference image is compared with projected EM images, and correlation values of the two images, and aligned projected EM images are output. The command used is mrcImageAutoRotationCorrelation. The parameters must be set up like below:
# For 2D STEP=12 ROTMIN=0 ROTMAX=359 nROT=`echo "" | awk 'BEGIN {printf 360 / $(STEP)}'`
STEP is an angle to be specified as in-plane rotation. If specified like below,
STEP=12
the image is rotated for each 12 degrees. In addition, the following is specified as a search range:
ROTMIN=0 ROTMAX=359
When the image is cut out, if direction has been determined to some extent, the range can be narrowed down.
Estimation of the projection angle of a projected image: .corinfo.3dinfo The values of image similarity are listed in *.corinfo. This command extracts some value seeming to be the optimal as *.3dinfo. In addition, projection angles are determined in the process of this command, and cut out as a file needed for 3D reconfiguration.
.corinfo.3dinfo: awk '/Cor/ { print $$7,$$16,$$2,$$3,$$4,$$5,$$9,$$11,$$12}' $*.corinfo | sort -r | sed -e s/.pad/.fit/ > $*.3dinfolst head -n 1 $*.3dinfolst | awk '{print $$2,$$3,$$4,$$5,$$6,$$1}' > $*.3dinfo ## ##18 -> ##7 ##
With regarding to this part of processing, a version in which a few bugs are hidden has already been delivered. So, when the sort command is executed, if the number of digits of the correlation values is not correct, a problem will occur. Change the command line like below:
.corinfo.3dinfo: awk '/Cor/ { printf("%15.6f %s %s %s %s %s %s %s %s¥n", $$7,$$16,$$2,$$3,$$4,$$5,$$9,$$11,$$12}' $*.corinfo | sort -r | sed -e s/.pad/.fit/ > $*.3dinfolst head -n 1 $*.3dinfolst | awk '{print $$2,$$3,$$4,$$5,$$6,$$1}' > $*.3dinfo ## ##18 -> ##7 ##
This command estimates the projection angle of the reference image showing the best correlation value as the estimated projection angle according to the specified correlation value. It is also possible to estimate a bit more precious projection angle using this corinfo file. By approximating the projection angle with the Gauss function using the neighboring values showing high correlation, it is also possible to estimate that angle more finely than the step width of the angle applied to searching.
Creation of a projection image set: .3dinfo.3dlst Hereafter, 3dlst is created from all 3dinfo files. The simplest method is as follows:
.3dinfo.3dlst: cat $*.3dinfo >> $(TARGET).3dlst
In this case, all information is used. In practice, only good correlation values are selected, or moving distance is limited. It is also possible to change or select the image used for 3D reconfiguration between the one created by moving roi or the one created by moving fit. The following is an example of the image list selected by doing so.
#####Select Particles#####
.corinfo.corinfo2: awk '/Cor/ { print $$18,$$16,$$2,$$3,$$4,$$5,$$9,$$11,$$12}' $*.corinfo | sort -r | sed -e 's/.roi/.fit/' > $*.corinfolst head -n 1 $*.corinfolst | awk '{if($(PAD_W)-$$8>=$$8&&$(PAD_H)-$$9>=$$9) print $$1,$$2,$$3,$$4,$$5,$$6,$$7,0+$$8,0+$$9;else if($(PAD_W)-$$8>=$$8) print $$1,$$2,$$3,$$4,$$5,$$6,$$7,0+$$8,$(PAD_H)-$$9;else if($(PAD_H)-$$9>=$$9) print $$1,$$2,$$3,$$4,$$5,$$6,$$7,$(PAD_W)-$$8,0+$$9;else print $$1,$$2,$$3,$$4,$$5,$$6,$$7,$(PAD_W)-$$8,$(PAD_H)-$$9}' > $*.corinfo2 .corinfo2.parcorinfolst: cat $*.corinfo2 >> $(TARGET).parcorinfolst CorinfoSort:: awk '{ print $$1,$$2,$$3,$$4,$$5,$$6,$$7,$$8,$$9}' $(TARGET).parcorinfolst | sort -r > $(TARGET).parcorinfolstsort SelectParticle:: awk '$$1 >= $(COR_THRESHOLD) || $$9 <= $(Y_SHIFT_THRESHOLD){print $$2,$$3,$$4,$$5,$$6,$$1}' $(TARGET).parcorinfolstsort > $(TARGET).3dlstselect
By replacing the $(TARGET).3dlstselect file with $(TARGET).3dlst, 3D reconfiguration will become possible.
VIII-3.(4) Makefile used for the exercise
The Makefile used for the exercise is as follows:
# # Eos for 3D reconstruction from 2D rois using 3D initail reference # .SUFFIXES: .3d .ref3d .ref2d .pad .padtmp .mask .maskfit .fit .fitmask .roi .corinfo .3dinfo .3dlst .3d .3dsm .3dsmCenter .ds6 .3dsmds6 .corinfo2 .corinfolst .cormap .parcorinfolst .parcorinfolstsort .gnuplotcorinfolst .3dlstselect .monlst .mon SHELL=/bin/bash -include ROIs -include CORINFOs -include PADs -include 3DINFOs -include 3DLIST -include CORINFO2s # # Initial Reference Model INITIAL=initial # Target structure name TARGET=all
# Simultaneous Jobs for AutoRotaionCorrelation JOP_NUM=-j 3
# # Search Area for 3D # ROTMODE=YOYS # Rot1 ROT1MIN=0 ROT1MAX=359 ROT1D=30 nRot1=12 # ROT2MIN=0 ROT2MAX=359 ROT2D=30 nRot2=12 # Rot3 ROT3MIN=0 ROT3MAX=0 ROT3D=30 nRot3=1 # For 2D STEP=12 ROTMIN=0 ROTMAX=359 nROT=`echo "" | awk 'BEGIN {printf 360 / $(STEP)}'` # Selection Step COR_THRESHOLD=0.4 Y_SHIFT_THRESHOLD=20 # Pad size for 2D PAD_W=64 PAD_H=64 ### ### Initial model from PDB ### REFSOURCE=121p-shift DELTA=2.5 SIZE_X=64 SIZE_Y=64 SIZE_Z=64 START_X=`awk 'BEGIN { print -1*$(DELTA)*$(SIZE_X)/2}'` START_Y=`awk 'BEGIN { print -1*$(DELTA)*$(SIZE_Y)/2}'` START_Z=`awk 'BEGIN { print -1*$(DELTA)*$(SIZE_Z)/2}'` SNRATIO=3 ############################### Help:: @echo "----- Makefile for Single Particle with Reference" @echo "1. Prepare files from EM images, which has a suffix of roi " @echo "2. Prepare an initial model as $(INITIAL).3d " @echo "3. make First" @echo "4. make All" @echo "If you want to repeat refinement again, " @echo " 1. make Again" @echo " 2. make All" @echo "" @echo "--- If you want to create an initail model from PDB ---" @echo " make InitialModel" @echo "" @echo "--- If you want to create an test from the initial model ---" @echo " make TestData" pad:$(ROIs:.roi=.pad) corinfo:$(PADs:.pad=.corinfo) 3dinfo:$(CORINFOs:.corinfo=.3dinfo) fit:$(ROIs:.roi=.fit) 3dlst:$(3DINFOs:.3dinfo=.3dlst) 3d:$(3DLIST:.3dlst=.3d) corinfo2:$(CORINFOs:.corinfo=.corinfo2) parcorinfolst:$(CORINFO2s:.corinfo2=.parcorinfolst) # # # InitialModel:: pdb2mrc -i $(REFSOURCE).pdb -o $(INITIAL).3d -nx $(SIZE_X) -ny $(SIZE_Y) -nz $(SIZE_Z) -Sx $(START_X) -Sy $(START_Y) -Sz $(START_Z) -dx $(DELTA) -dy $(DELTA) -dz $(DELTA) -m 1 TestData:: @for (( rot1 = $(ROT1MIN); rot1 <= $(ROT1MAX); rot1 += $(ROT1D)*2 )) \ do \ for (( rot2 = $(ROT2MIN); rot2 <= $(ROT2MAX); rot2 += $(ROT2D)*2 )) \ do \ for (( rot3 = $(ROT3MIN); rot3 <= $(ROT3MAX); rot3 += $(ROT3D)*2 )) \ do \ echo $$rot1, $$rot2, $$rot3; \ mrc3Dto2D -i $(INITIAL).3d -o $(REFSOURCE)-$$rot1-$$rot2-$$rot3.roi \ -InterpolationMode 2 -EulerMode $(ROTMODE) \ -Rot1 $$rot1 $$rot1 1 \ -Rot2 $$rot2 $$rot2 1 \ -Rot3 $$rot3 $$rot3 1 ; \ done \ done \ done TestDataNoise:: @for (( rot1 = $(ROT1MIN); rot1 <= $(ROT1MAX); rot1 += $(ROT1D)*2 )) \ do \ for (( rot2 = $(ROT2MIN); rot2 <= $(ROT2MAX); rot2 += $(ROT2D)*2 )) \ do \ for (( rot3 = $(ROT3MIN); rot3 <= $(ROT3MAX); rot3 += $(ROT3D)*2 )) \ do \ echo $$rot1, $$rot2, $$rot3; \ name="$(REFSOURCE)-noise-$$rot1-$$rot2-$$rot3.roi"; \ mrc3Dto2D -i $(INITIAL).3d -o $$name.noNoise \ -InterpolationMode 2 -EulerMode $(ROTMODE) \ -Rot1 $$rot1 $$rot1 1 \ -Rot2 $$rot2 $$rot2 1 \ -Rot3 $$rot3 $$rot3 1 ; \ mrcImageNoiseAdd -i $$name.noNoise -o $$name -SN $(SNRATIO) ; \ rm $$name.noNoise ; \ done \ done \ done ############## ROIs:: touch ROIs echo "ROIs=\\" > ROIs #find -name "*.roi" -type f | sed s/..// | xargs ls -1 | sed s/roi/roi\\\\/ >> ROIs ls -1 *.roi | sed s/roi/roi\\\\/ >> ROIs echo "" >> ROIs CORINFOs:: touch CORINFOs echo "CORINFOs=\\" > CORINFOs #find -name "*.corinfo" -type f | sed s/..// | xargs ls -1 | sed s/corinfo/corinfo\\\\/ >> CORINFOs ls -1 *.corinfo | sed s/corinfo/corinfo\\\\/ >> CORINFOs echo "" >> CORINFOs PADs::
touch PADs
echo "PADs=\\" > PADs #find -name "*.pad" -type f | sed s/..// | xargs ls -1 | sed s/pad/pad\\\\/ >> PADs ls -1 *.pad | sed s/pad/pad\\\\/ >> PADs echo "" >> PADs 3DINFOs:: touch 3DINFOs echo "3DINFOs=\\" > 3DINFOs #find -name "*.3dinfo" -type f | sed s/..// | xargs ls -1 | sed s/3dinfo/3dinfo\\\\/ >> 3DINFOs ls -1 *.3dinfo | sed s/3dinfo/3dinfo\\\\/ >> 3DINFOs echo "" >> 3DINFOs 3DLIST:: touch 3DLIST
echo "3DLIST=\\" > 3DLIST
#find -name "*.3dlst" -type f | sed s/..// | xargs ls -1 | sed s/3dlst/3dlst\\\\/ >> 3DLIST ls -1 *.3dlst | sed s/3dlst/3dlst\\\\/ >> 3DLIST echo "" >> 3DLIST CORINFO2s:: touch CORINFO2s echo "CORINFO2s=\\" > CORINFO2s #find -name "*.corinfo2" -type f | sed s/..// | xargs ls -1 | sed s/corinfo2/corinfo2\\\\/ >> CORINFO2s ls -1 *.corinfo2 | sed s/corinfo2/corinfo2\\\\/ >> CORINFO2s echo "" >> CORINFO2s ############## First:: cp $(INITIAL).3d $(TARGET).ref3d touch *.roi All:: make $(TARGET).ref2d; make ROIs; make $(JOP_NUM) pad; make PADs; make $(JOP_NUM) corinfo; make CORINFOs; make $(JOP_NUM) 3dinfo; make 3DINFOs; make $(JOP_NUM) 3dlst; make 3DLIST; make $(TARGET).3d; Again:: touch *.pad cp $(TARGET).3d $(TARGET).ref3d; TMPNAME=`date +%y%m%d%H%M%S`; \ mv $(TARGET).3dlst $(TARGET).3dlst.$$TMPNAME; \ mv $(TARGET).3d $(TARGET).3d.$$TMPNAME; \ mv $(TARGET).3dsm $(TARGET).3dsm.$$TMPNAME; \ mv $(TARGET).ds6 $(TARGET).ds6.$$TMPNAME; \ mv $(TARGET).3dsmds6 $(TARGET).3dsmds6.$$TMPNAME; #### Prepare Reference #### .ref3d.ref2d: mrc3Dto2D -i $(TARGET).ref3d -o $(TARGET).ref2d -EulerMode $(ROTMODE) -InterpolationMode 2 -Rot2 $(ROT2MIN) $(ROT2MAX) $(ROT2D) -Rot1 $(ROT1MIN) $(ROT1MAX) $(ROT1D) -Rot3 $(ROT3MIN) $(ROT3MAX) $(ROT3D) -m 0 .roi.pad: mrcImageWindowing -i $*.roi -o $*.mask -W 0.1 0.0 0.05 0.0 -m 18 mrcImagePad -i $*.mask -o $*.padtmp -W $(PAD_W) -H $(PAD_H) -m 3 mrcImageWindowing -i $*.padtmp -o $*.pad -W 0.1 0.0 0.1 0.0 -m 2 .pad.corinfo: mrcImageAutoRotationCorrelation -i $*.pad -r $(TARGET).ref2d -O $*.corinfo -fit $*.fit -cor $*.cormap -n $(nRot) -m 18 -range $(ROTMIN) $(ROTMAX) -Iter 2 -nRot2 $(nRot2) -nRot1 $(nRot1) -nRot3 $(nRot3) 2> /dev/null ############################################################################## .corinfo.3dinfo: awk '/Cor/ { print $$7,$$16,$$2,$$3,$$4,$$5,$$9,$$11,$$12}' $*.corinfo | sort -r | sed -e s/.pad/.fit/ > $*.3dinfolst head -n 1 $*.3dinfolst | awk '{print $$2,$$3,$$4,$$5,$$6,$$1}' > $*.3dinfo ## ##18 -> ##7 ## .3dinfo.3dlst: cat $*.3dinfo >> $(TARGET).3dlst .3dlst.3d: mrc2Dto3D -I $(TARGET).3dlst -o $(TARGET).3d -InterpolationMode 2 -Double -DoubleCounter $(TARGET).3dcounter -CounterThreshold 0.5 -m 1 -WeightMode 2 .3d.3dsm: mrcImageSmoothing -i $(TARGET).3d -o $(TARGET).3dsm -m 1 -M 1 -r 3 .3dsm.3dsmCenter: mrcImageCenterGet -i $(TARGET).3dsm -o $(TARGET).3dsmCenter -Nx 128 -Ny 128 -Nz 128 .3dsmCenter.ds6: mrc2map -i $(TARGET).3dsmCenter -o $(TARGET).ds6 -m 3 .3d.ds6: mrc2map -i $*.3d -o $*.ds6 -m 0 #####Select Particles##### .corinfo.corinfo2: awk '/Cor/ { print $$18,$$16,$$2,$$3,$$4,$$5,$$9,$$11,$$12}' $*.corinfo | sort -r | sed -e 's/.roi/.fit/' > $*.corinfolst head -n 1 $*.corinfolst | awk '{if($(PAD_W)-$$8>=$$8&&$(PAD_H)-$$9>=$$9) print $$1,$$2,$$3,$$4,$$5,$$6,$$7,0+$$8,0+$$9;else if($(PAD_W)-$$8>=$$8) print $$1,$$2,$$3,$$4,$$5,$$6,$$7,0+$$8,$(PAD_H)-$$9;else if($(PAD_H)-$$9>=$$9) print $$1,$$2,$$3,$$4,$$5,$$6,$$7,$(PAD_W)-$$8,0+$$9;else print $$1,$$2,$$3,$$4,$$5,$$6,$$7,$(PAD_W)-$$8,$(PAD_H)-$$9}' > $*.corinfo2
.corinfo2.parcorinfolst: cat $*.corinfo2 >> $(TARGET).parcorinfolst CorinfoSort:: awk '{ print $$1,$$2,$$3,$$4,$$5,$$6,$$7,$$8,$$9}' $(TARGET).parcorinfolst | sort -r > $(TARGET).parcorinfolstsort SelectParticle:: awk '$$1 >= $(COR_THRESHOLD) || $$9 <= $(Y_SHIFT_THRESHOLD){print $$2,$$3,$$4,$$5,$$6,$$1}' $(TARGET).parcorinfolstsort > $(TARGET).3dlstselect #####Check alignment##### GnuplotCorinfo:: awk '{ print $$1,$$9}' $(TARGET).parcorinfolst | sort -r > $(TARGET).gnuplotcorinfolst #################################################### ##gnuplot ##set out "Corinfo.ps" ##plot "$(TARGET).gnuplotcorinfolst" ################################################### GnuplotAngle:: awk '{ print $$4,$$5}' $(TARGET).parcorinfolst > $(TARGET).gnuplotanglelst ###################################################### ##gnuplot ##set out "Angle.ps" ##plot "$(TARGET).gnuplotanglelst" ###################################################### GnuplotShift:: awk '{ print $$4,$$5}' $(TARGET).parcorinfolst > $(TARGET).gnuplotshiftlst ###################################################### ##gnuplot ##set out "Shift.ps" ##plot "$(TARGET).gnuplotshiftlst" ###################################################### MontageList:: awk '{ print $$1}' $(TARGET).3dlst > $(TARGET).monlst mrcImageMontageCreate -i $(TARGET).monlst -o $(TARGET).mon -nx 15 -ny 10