Computers : Data processing General nodes¶

Data Policy¶

===============================================================================

Using general nodes follow the below steps

• Step 1 : login to ssh <username>@lnx201.classe.cornell.edu

• Step 2: qrsh -q interactive.q -l mem_free=350G

  Or if you need more core

• Step 2: qrsh -q interactive.q -l mem_free=350G -pe sge_pe 20

• Step 3: source /nfs/chess/sw/anaconda3_jpcr/bin/activate
• Step 4: qstat
• Step 5 : ls
• Step 6: cd /nfs/chess/id4baux/2023-2/codebase_for_users/
• Step 7: Follow Step II above

If you are doing stacking from general node, you need to change the permission setup

Giving permission from your own account to chess_id4b to do rest of the processing follow the below steps ¶

• Step 1: If you are using general cluster, then follow the steps from your account

  a) Go to the desired path b) chmod -R 777 (foldername)

The script take all the detector's raw .cbf files and performed calibration. You can the get the stack_em_all_cbf_2022.py.

Step 1 :  the `python stack_em_all_cbf_2022.py` python script
Modify : 
        i) Project name : 
                        proj_name = "sarker-000-a/" (#project name)
        ii) Provide the .poni filename
                        calibfile = "/nfs/chess/id4baux/"+run_cycle+"+proj_name+"/calibration/ceria37keV.poni" 
        iii)Provide .edf filename
                        maskfile = "/nfs/chess/id4baux/"+run_cycle+"+proj_name+"/calibration/mask.edf"

Step 2 : Run python code
    Usage:
    python stack_em_all_cbf_2022.py    chem_formula sample_id spec_scan_num temperature

    Example: 
    >>>python stack_em_all_cbf_2022.py CoTiO3 sample 11 50

Step 3 : Check data 
    After stacking is done the processed data will be avilable in id4baux folder

You can create bash script to run the stacking

    #!  /bin/bash
    python stack_em_all_cbf_2022.py CoTiO3 sample 11 50
    python stack_em_all_cbf_2022.py CoTiO3 sample 12 50
    python stack_em_all_cbf_2022.py CoTiO3 sample 13 50

===========================================================================

simple_peakfinder.py script

Step 1: After stacking is done, take the whole stack from the id4aux data folder and run simple_peakfinder.py code

    Usage: 
    python simple_peakfinder.py /full/path/to/stack/ 0.95

    Example: 
    >>> python simple_peakfinder.py /nfs/chess/id4baux/2022-3/sarker-0000-a/CoTiO3/sample/300/ 0.95

Would consider anything above 95% of max counts as a Bragg peak. You can change the thereshold as needed. Check the number of peaks, you find after running the code. In between 200-3000 is a good number.

===========================================================================

auto_ormfinder.py script

Usage: 
python auto_ormfinder.py */full/path/to/stack/ peakfilename.npy

Example: 
>>>python auto_ormfinder.py /nfs/chess/id4baux/2022-3/sarker-0000-a/CoTiO3/sample/300/ peaklist1.npy

===========================================================================

Pil6M_HKLConv_3D_2022.py python script (edit to point to project, ormatrix, and define HKL grid)

Step 1: Edit the script
    i) Change the project name
                projd = "/nfs/chess/id4baux/2022-3/sarker-0000-a"
                sampldi = "/CoTiO3/sample/" 
    ii) Change the sample name:
                sampldi = "/CoTiO3/sample/" 
    iii) Change the ormatix 
                ormat = nxload(stackhklpath+"ormatrix_v1.nxs")
    iv) Define the hkl space to histogram
                H= np.arrange (-5.1, 5.1, 0.01)
                K= np.arrange (-5.1, 5.1, 0.01)
                L= np.arrange (-5.1, 5.1, 0.01)

Step 2: 
Usage:
python  Pil6M_HKLConv_3D_2022.py temperature

Example:
python Pil6M_HKLConv_3D_2022.py 50

• Step A: Do the stacking (follow STEP I above)
• Step B: Copy the orientation matrix from processed data
• Step C: Run STEP V

===========================================================================

To tunnel the jupyter notebook from a node to the browser on the station computer, try this:

• 1) setup your environment and start jupyter on the node (ie lnx306 or whatever) with jupyter notebook --no-browser --port=8888 --ip="172.16.3.6"
• 2) from the station computer, open a new terminal and do ssh -N -f -L localhost:8888:lnx306.classe.cornell.edu:8888
• 3) on the station computer, browse to localhost:8888 and enter the token as before (edited)

Important links for more details ¶

• Remote access to Linux machines
• Access to nomachine
• Data transfer Globus
• CHESS computing farm access
• CHESS jupyterhub
• CLASSE-IT Service Requests