LANL CESM Exercise1
Lab exercise: Running CCSM
In this exercise you will create an "IG" case (active land and ice sheet models forced by atmospheric data) and then configure, modify, build, and run the code.
First log onto mapache. (If you are already logged on, please log out and back in to clear your shell variables.) Typically you would type this:
> ssh -t -X wtrw.lanl.gov (Enter your Cryptocard password) > ssh mp-fe1 (or ssh mp-fe2)
But in the Encantado classroom, you will need to bring up Putty and follow the directions in the handout, "Logging into Mapache from Encantado Classroom Terminals".
Go to your working directory (e.g., /usr/projects/climate/username or /usr/projects/workroom/usrname) and copy cesm1_0_2 to this directory:
> cp -rf /usr/projects/cesm/cesm1_0_2 .
The scripts in this version of the model have already been set to the appropriate values for mapache, so you do not need to import them by hand.
You should now have a directory called cesm1_0_2 with subdirectories called models and scripts.
Let's first look in the models directory:
> cd cesm1_0_2/models > ls
The models directory contains several subdirectories:
- atm = atmosphere
- ocn = ocean
- lnd = land
- ice = sea ice
- glc = ice sheet (CISM)
- drv = driver (includes coupler modules)
- csm_share = shared code
- utils = utilities
NOTE: In CESM, ice refers to the sea ice model. The ice-sheet model is not in the ice directory, but in the glc directory. Let's look at glc:
> cd glc > ls
The active ice sheet model is in the cism (Community Ice Sheet Model) directory. The directories sglc and xglc contain the stub and dead components.
Go to the CISM directory:
> cd cism > ls
The source code resides in several subdirectories:
- source_glimmer-cism = Glimmer-CISM code
- source_slap = code that is part of the SLAP solver library
- source_glc = code needed to link Glimmer-CISM to the CESM coupler
- drivers = more code associated with the coupler
The other CESM models (atm, lnd, ocn, and ice) are structured in a similar way.
Create a case
There are many ways to configure and run CESM. For example, you can run various combinations of active, data, and stub components at different grid resolutions. A particular combination of components and grids is called a case.
You will now create a case that consists of an active land component (CLM, the Community Land Model), an active ice-sheet component (CISM, the Community Ice Sheet Model), a data atmosphere component, and stub ocean and sea ice components. The data atmosphere component is from an NCEP reanalysis at T62 (~1.5 deg) resolution. The land model will (among many other things) compute a surface mass balance that is used to force the ice sheet model.
To create a case, go to the scripts directory.
> cd ~/your-home-dir/cesm1_0_2/scripts > ls
We will run a script called create_newcase. (You have done this already if you took part in other exercises.) Type the following to create an IG case for mapache:
> ./create_newcase -case IGcase -mach mapache -res f19_g16 -compset IG
- IGcase is the name of the case, which you can call whatever you want
- mach = the name of the machine where you are running the model
- res = resolution
- 1.9x2.5 = 1.9 x 2.5 degree grid for the atmosphere and land models
- 0.9x1.25 = 0.9 x 1.25 degree grid for the atmosphere and land models
- T31 = spectral T31 grid (relatively coarse) for the atmosphere and land models
- gx1v6 = 1 degree grid, version 6 for the ocean and sea ice models
- gx3v7 = 3 degree grid, version 7 for the ocean and sea ice models
- f19_g16 is shorthand for the 1.9x2.5 atmosphere/land grid combined with the gx1v6 ocean/ice grid.
- compset = set of active physical components, for example:
- A: all data models; no active physical components
- I: active land
- IG: active land and ice sheet
- F: active land and atmosphere
- FG: active land, atmosphere, and ice sheet
- B: active land, atmosphere, ocean, and sea ice
- BG: active land, atmosphere, ocean, sea ice, and ice sheet
Many other compsets are allowed. For examples, please see the CESM User's Guide: http://www.cesm.ucar.edu/models/cesm1.0/cesm/
After creating a case using the command above, you should see on your screen a message like this:
> Successfully created the case for mapache
Since the IG case does not have an active atmosphere or ocean model, it will run relatively quickly. At an atmosphere/land resolution of 1.9 x 2.5 degrees, it will take about 20 minutes to run one calendar year on 16 cores. So we will have time to launch a multiyear climate experiment and look at the results before the end of the afternoon.
Configure and build the model
Next, you will configure and build CESM for your IG case. Go to your new case directory:
> cd IGcase > ls
There are several files with the suffix xml. These files are used to set various model options and parameters.
Look at env_conf.xml:
> less env_conf.xml
For ice-sheet modeling, a setting of interest is $GLC_GRID. The default value is gland20, which refers to the 20-km Greenland grid. If you wanted to run with a finer grid, you would change this to gland10 or gland5. But for this exercise, the 20-km grid works fine.
Next, look at env_mach_pes.xml:
> less env_mach_pes.xml
This file determines how many processors are assigned to each model component. For our case the processors will be assigned to each component sequentially during each time step. In other words, all the processors available will be used first to run the atmosphere model, then to run the land model, and so on until the end of the time step.
The default number of processors is 64. Since we have a limited number of processors available, we will change env_mach_pes.xml to assign 16 processors sequentially to each model.
Although xml files can be edited manually, it is safer to use the xmlchange command. Type the following to make changes in env_mach_pes.xml:
> ./xmlchange -file env_mach_pes.xml -id NTASKS_ATM -val 16 > ./xmlchange -file env_mach_pes.xml -id NTASKS_LND -val 16 > ./xmlchange -file env_mach_pes.xml -id NTASKS_ICE -val 16 > ./xmlchange -file env_mach_pes.xml -id NTASKS_OCN -val 16 > ./xmlchange -file env_mach_pes.xml -id NTASKS_CPL -val 16
These changes must be made before you configure the code. Currently, the glc component runs on only one processor, so you should not change the value of NTASKS_GLC.
Now configure the code by typing this:
> ./configure -case
Look again in the case directory:
If the configure is successful, you will have several new scripts:
Before building the code, let's make things more interesting by making a small change. Your IG case is configured to run with atmospheric data for the year 2003 from an NCEP reanalysis. What if the surface air temperature were higher or lower than the values in the reanalysis?
To find out, you can change a single line of code in CLM. The safest way to change a file is to make a copy of the file in one of your SourceMods directories and change it there, leaving the original version unchanged. In this way you can easily keep track of your code changes.
The file you will modify is called clm_driverInitMod.F90, and it is part of the CLM source code. Type the following to copy it to the CLM SourceMods directory:
> cp ~/your-home-dir/cesm1_0_2/models/lnd/clm/src/biogeophys/clm_driverInitMod.F90 SourceMods/src.clm/
Now go to the CLM SourceMods directory:
> cd SourceMods/src.clm > ls
There will be a single file, clm_driverInitMod.F90. When the code is built, any files that are in this directory will automatically replace files of the same name in the CLM source code directory (in this case, ~/your-home-dir/cesm1_0_2/models/lnd/clm/src/biogeophys/clm_driverInitMod.F90).
Using your preferred text editor (e.g., vi or emacs), you will edit the version of clm_driverInitMod.F90 in the CLM SourceMods directory. Find this line of code:
> tbot_c = tbot_g-lapse_glcmec*(hsurf_c-hsurf_g) ! sfc temp for column
This part of the code sets the surface air temperature, tbot_c, for glacier columns. Recall that in each grid cell containing glacial ice, the glaciated area is divided into several columns, each with a different elevation. The surface air temperature at the mean gridcell elevation is tbot_g. For each column, tbot_g is adjusted by adding a term proportional to the elevation difference between the mean gridcell elevation hsurf_g and the local column elevation hsurf_c.
You will change this line to something like this:
> tbot_c = tbot_g-lapse_glcmec*(hsurf_c-hsurf_g) + 1.0 ! sfc temp for column, plus one degree
When you are ready to make this change, please tell one of the instructors. We will make sure that each group chooses a different value for the temperature change.
In this way you will have a crude version of a global warming (or cooling) simulation. Later we will see how this temperature change affects the surface mass balance of the Greenland ice sheet.
Before building the code, you will need to make one other change. Go to the CISM SourceMods directory:
> cd ../src.cism
And copy two modified files into this directory:
> cp /usr/projects/cesm/src/cism/*90 . > ls
You should now see files called glint_mbal.F90 and glint_smb.F90. These files contain minor fixes to allow the code to build with the pgi compiler. (This bug will be fixed in future CESM releases.)
Once you've made your code changes in SourceMods/src.clm, go back to the case directory and build the model:
> cd ../.. > ls > ./IGcase.mapache.build
If the build fails, you will get an error message pointing you to a log file in another directory. If you look at the bottom of that log file, you will usually see what went wrong.
The build will take a few minutes. If the code builds successfully, you will see on your screen a message like this:
> CCSM BUILDEXE SCRIPT HAS FINISHED SUCCESSFULLY
Run the model
Now you are ready to run CESM. Go to your case directory if you are not there already:
> cd ~/your-home-dir/cesm1_0_2/scripts/IGcase
First look at the file env_run.xml:
> less env_run.xml
This file contains a number of options that you may want to change after you build the code, but before you submit a run.
For now, the most important options are STOP_N and STOP_OPTION, which determine the length of the model run. The default values are STOP_OPTION = ndays and STOPN = 5. This means the code will run for 5 days--just long enough to make sure nothing is seriously broken. It is a good idea to try a 5-day run before attempting a longer run.
Go ahead and submit the batch job:
If all is well, you will get a message like this:
> check_case OK > jobnumber
To check the job status, type this:
> showq -w user=username
When the job is running, you should see on your screen a message like this:
> active jobs------------------------ > JOBID USERNAME STATE PROCS REMAINING STARTTIME > 6596 lipscomb Running 16 00:04:58 Wed Apr 27 09:44:17
If no jobs (active, eligible, or blocked) are listed under your user name, then your job is probably done.
If all goes well, the job will finish within one or two minutes after starting. To see if the job completed successfully, look at the output file in your case directory:
> less log.o
If the job finished successfully, you will see some lines like this near the end:
> CCSM PRESTAGE SCRIPT HAS FINISHED SUCCESSFULLY > Wed Apr 27 09:54:01 MDT 2011 -- CSM EXECUTION BEGINS HERE > Wed Apr 27 09:54:33 MDT 2011 -- CSM EXECUTION HAS FINISHED > (seq_mct_drv): =============== SUCCESSFUL TERMINATION OF CPL7-CCSM =============== > Archiving ccsm output to /scratch2/lipscomb/CESM/archive/IGcase_p5 > Calling the short-term archiving script st_archive.sh
Some log files will have been written to your logs directory. Let's look at them:
> cd logs > ls
These files have a suffix .gz, which means they have been compressed. Uncompress the glc log file and take a look:
> gunzip glc.log.110427-xxxxxx.gz > less glc.log.110427-xxxxxx
The 12-digit number is a date stamp of the form yymmdd-hhmmss. Your file will have a different date stamp.
Scroll through the log file. You will see information about the Glimmer-CISM input parameters, followed by some model diagnostics that are written out during the run. If the run finished successfully, you will see this near the end:
> Successful completion of glc model
There are similar log files for the coupler, land, and atmosphere components (cpl, lnd, and atm). The logfile with the ccsm prefix combines diagnostics from each component.
Now return to your case directory:
> cd ..
Once you have finished a 5-day run, you are ready to try a longer run. We will set the model to run for a year, then resubmit itself four times until it has completed a 5-year run. In the file env_run.xml, change the values of STOP_OPTION and STOP_N:
> xmlchange -file env_run.xml -id RESUBMIT -val 4 > xmlchange -file env_run.xml -id STOP_OPTION -val nyears > xmlchange -file env_run.xml -id STOP_N -val 1
Then submit the batch job as before:
After a minute or so, check the job status:
> showq -w user=username
With luck, everyone's job will be under way.
A 5-year run will take about 100 minutes to complete. We will look at some results later, after doing a Glimmer-CISM exercise.