What is GGIR?

GGIR is an R-package to process multi-day raw accelerometer data for physical activity and sleep research. GGIR will write all output files into two sub-directories of ./meta and ./results. GGIR is increasingly being used by a number of academic institutes across the world.

What is postGGIR?

postGGIR is an R-package to data processing after running GGIR for accelerometer data. In detail, all necessary R/Rmd/shell files were generated for data processing after running GGIR for accelerometer data. Then in part 1, all csv files in the GGIR output directory were read, transformed and then merged. In part 2, the GGIR output files were checked and summarized in one excel sheet. In part 3, the merged data was cleaned according to the number of valid hours on each night and the number of valid days for each subject. In part 4, the cleaned activity data was imputed by the average ENMO over all the valid days for each subject. Finally, a comprehensive report of data processing was created using Rmarkdown, and the report includes few explortatory plots and multiple commonly used features extracted from minute level actigraphy data in part 5-7. This vignette provides a general introduction to postGGIR.

Software Architecture

The R package postGGIR has been released with an open-source GPL-3 license on CRAN, and postGGIR can run on Windows and Linux. Parallel computing in Linux is recommended due to the memory requirements associated with reading in multiple of the large data files. The package contains one primary function for users which, when run, generates all necessary R/R Markdown/shell executable files for data processing after running GGIR for accelerometer data; load, read, transform and merge long activity data; examine and summarize GGIR outputs; clean the merged activity data according to the number of valid hours per night and the number of valid days per subject; activity data imputation by taking the average across the valid days for each subject; build a comprehensive report of data processing and exploratory plots; extract multiple commonly used features and study feature structure by the covariance decomposition. Figure 1 presents a flowchart for each step in this process which is described in greater detail below. The procedure, R functions, inputs, and outputs are all described in this package vignette. In addition, more documentation and example data could be found in postGGIR repository on GitHub (URL: https://github.com/dora201888/postGGIR).

Software Dependencies

All postGGIR code is written in R and reports generated in R Markdown. The R packages ActFrag and ActCR are used for the calculation of certain physical activity and circadian rhythmicity features. The R package r.jive is used to perform the feature interaction analysis and to study the joint and individual variation structure by JIVE.

Install R and RStudio

Download and install R

Download and install RStudio (optional, but recommended)

Download GGIR with its dependencies, you can do this with one command from the console command line:

install.packages("postGGIR", dependencies = TRUE)

Prepare folder structure

1. folder of .bin files for GGIR or a file listing all .bin files
   • R program will check the missing in the GGIR output by comparing with all raw .bin files
2. foder of the GGIR output with two sub-folders
   • meta (./basic, ./csv, etc)
     
   • results (partsummary.csv)

Create a template shell script of postGGIR

library(postGGIR)
create.postGGIR()

The function will create a template shell script of postGGIR in the current directory, names as STUDYNAME_part0.maincall.R.

cat STUDYNAME_part0.maincall.R

options(width=2000) 
argv = commandArgs(TRUE);  
print(argv) 
print(paste("length=",length(argv),sep=""))  
mode<-as.numeric(argv[1])  
print(c("mode =", mode))
# (Note) Please remove the above lines if you are running this within R console 
#        instead of submitting jobs to a cluster.
 
#########################################################################   
# (user-define 1) you need to redefine this according different study!!!!
######################################################################### 
# example 1 
filename2id.1<-function(x)  unlist(strsplit(y1,"\\."))[1] 
 
#  example 2 (use csv file =c("filename","ggirID")) 
filename2id.2<-function(x) {
  d<-read.csv("./postGGIR/inst/example/filename2id.csv",head=1,stringsAsFactors=F)
  y1<-which(d[,"filename"]==x)
  if (length(y1)==0) stop(paste("Missing ",x," in filename2id.csv file",sep=""))
  if (length(y1)>=1) y2<-d[y1[1],"newID"] 
  return(as.character(y2))
} 


#########################################################################  
#  main call
######################################################################### 
  
call.afterggir<-function(mode,filename2id=filename2id.1){ 

library(postGGIR) 
#########################################################################  
# (user-define 2) Fill in parameters of your ggir output
########################################################################## 
  
currentdir = 
studyname =
bindir = 
outputdir =  
setwd(currentdir) 

rmDup=FALSE   # keep all subjects in postGGIR
PA.threshold=c(50,100,400)
part5FN="WW_L50M125V500_T5A5" 
epochIn = 5
epochOut = 5
flag.epochOut = 60
use.cluster = FALSE
log.multiplier = 9250
QCdays.alpha = 7
QChours.alpha = 16 
useIDs.FN<-NULL 
Rversion="R" 
desiredtz="US/Eastern" 
RemoveDaySleeper=FALSE 
part5FN=part5FN,
NfileEachBundle=20 
trace=FALSE
#########################################################################  
#   remove duplicate sample IDs for plotting and feature extraction 
######################################################################### 
if (mode==3 & rmDup){
# step 1: read ./summary/*remove_temp.csv file (output of mode=2)
keep.last<-TRUE #keep the latest visit for each sample
sumdir<-paste(currentdir,"/summary",sep="")  
setwd(sumdir)  
inFN<-paste(studyname,"_samples_remove_temp.csv",sep="")
useIDs.FN<-paste(sumdir,"/",studyname,"_samples_remove.csv",sep="") 

#########################################################################  
# (user-define 3 as rmDup=TRUE)  create useIDs.FN file
######################################################################### 
# step 2: create the ./summary/*remove.csv file manually or by R commands
d<-read.csv(inFN,head=1,stringsAsFactors=F)
d<-d[order(d[,"Date"]),]
d<-d[order(d[,"newID"]),]
d[which(is.na(d[,"newID"])),]
S<-duplicated(d[,"newID"],fromLast=keep.last) #keep the last copy for nccr
d[S,"duplicate"]<-"remove"
write.csv(d,file=useIDs.FN,row.names=F)

} 

#########################################################################  
#   call afterggir
######################################################################### 

setwd(currentdir)  
afterggir(mode=mode,
          useIDs.FN=useIDs.FN,
          currentdir=currentdir,
          studyname=studyname,
          bindir=bindir,
          outputdir=outputdir,
          epochIn=epochIn,
          epochOut=epochOut,
          flag.epochOut=flag.epochOut,
          log.multiplier=log.multiplier,
          use.cluster=use.cluster,
          QCdays.alpha=QCdays.alpha,
          QChours.alpha=QChours.alpha,
          QCnights.feature.alpha=QCnights.feature.alpha, 
          Rversion=Rversion,
          filename2id=filename2id,
          PA.threshold=PA.threshold,
          desiredtz=desiredtz,
          RemoveDaySleeper=RemoveDaySleeper,
          part5FN=part5FN,
          NfileEachBundle=NfileEachBundle,
          trace=trace) 

} 
#########################################################################
          call.afterggir(mode)   
######################################################################### 

#   Note:   call.afterggir(mode)
#        mode = 0 : creat sw/Rmd file
#        mode = 1 : data transform using cluster or not
#        mode = 2 : summary
#        mode = 3 : clean 
#        mode = 4 : impu

Edit shell script

Three places were marked as “user-define” and need to be edited by user in the STUDYNAME_part0.maincall.R file. Please rename the file by replacing your real studyname after the edition.

Run R script

call.afterggir(mode,filename2id)   

Run script in a cluster

#!/bin/bash
#
#$ -cwd
#$ -j y
#$ -S /bin/bash
  source ~/.bash_profile

   cd /postGGIR/inst/example/afterGGIR; 
   module load R ; 
     R --no-save --no-restore --args  < studyname_ggir9s_postGGIR.pipeline.maincall.R  0
     R --no-save --no-restore --args  < studyname_ggir9s_postGGIR.pipeline.maincall.R  1
     R --no-save --no-restore --args  < studyname_ggir9s_postGGIR.pipeline.maincall.R  2
     R --no-save --no-restore --args  < studyname_ggir9s_postGGIR.pipeline.maincall.R  3
     R --no-save --no-restore --args  < studyname_ggir9s_postGGIR.pipeline.maincall.R  4 

     R -e "rmarkdown::render('part5_studyname_postGGIR.report.Rmd'   )" 
     R -e "rmarkdown::render('part6_studyname_postGGIR.nonwear.report.Rmd'   )" 
     R -e "rmarkdown::render('part7a_studyname_postGGIR_JIVE_1_somefeatures.Rmd'   )" 
     R -e "rmarkdown::render('part7b_studyname_postGGIR_JIVE_2_allfeatures.Rmd'   )" 
     R -e "rmarkdown::render('part7c_studyname_postGGIR_JIVE_3_excelReport.Rmd'   )" 

Part 1

The functions of part 1 read the activity data measured by Euclidian norm minus one (ENMO), a rotationally invariant measure of volume of acceleration, in long csv-spreadsheets and then the data was merged and transformed into a square matrix for all subjects. Depending on the time zone on which the devices were initialized, a day may have either 23 or 25 due to daylights savings time. On daylight savings crossovers, ENMO is averaged for duplicate timestamps between 1:00 AM and 2:00 AM, allowing for straightforward comparisons to standard 24-hour days. In addition to recording ENMO at each epoch, the angle of the z-axis (ANGLEZ) relative to the horizontal plane (degrees), used for estimating sleep periods, is merged and saved into an excel file. Additional data recorded by certain device, including the light mean, light peak, temperature mean, clipping score, and the Euclidian norm metric (EN) if available are merged with the ENMO data and saved under the ./data directory for part 1.

The activity data in the GGIR output is formatted in long csv-spreadsheets as follows,

Each row represents the corresponding ENMO and ANGLEZ values at a timestamp per 5 seconds epoch, which is specified by GGIR parameter (windowsizes) when running GGIR. After running part 1, the ENMO and ANGLEZ data are transformed into wide matrix in which each row represents 24 hours data for a day. For example, the ENMO data is formated as follows,

Finally, the data was merged for all days and all subjects.

Part 2

The functions of part 2 introduce descriptive variables of all accelerometer files in the GGIR output. In part 2, an excel file was output under the ./summary directory, which includes nine pages as follows, 1) List of files in the GGIR output (2) Summary of numbers of output files (3) List of duplicate IDs (4) ID errors (5) Number of valid days (6) Table of number of valid/missing days (7) Missing pattern (8) Frequency of the missing pattern (9) Description of all accelerometer files. Multiple plots were generated in a pdf file including the number of valid hours, days, missing pattern, etc. Technically, the raw accelerometer data was visited to obtain a complete input list for the purpose of examining missingness of GGIR output when user specified the path. Additionally, the summary results from GGIR output were also visited here to form a comprehensive data quality report in the part 2 of postGGIR.

Part 3 and Part 4

The functions of part 3 introduce ‘flag’ variables for data cleaning of the merged ENMO and ANGLEZ data. By default, days with more than 16 hours were marked as valid days and subjects with more than 7 valid days were marked as valid samples. User can set these two parameters in the call to the main function (QCdays.alpha = 7, QChours.alpha = 16). Further, the data can be aggregated into minute-level or hour-level data as required by users. In part 4, ENMO data during estimated non-wear periods were imputed by taking the subject-level mean over all the valid days for each subject at that time. All output is saved under the ./data directory. In the output file, the following description variables are included: (1) number of valid hours; (2) missing pattern for each subject; (3) non-wear time in minutes; (4) an indicator variable to indicate if the visit should be removed for having multiple visits for some subjects which might lead to invalidity of independent and identically distribution in statistics and (5) the number of missing values after imputation. When the number of missing values after imputation is not zero, it means the activity data is missing on same timestamps among all days and therefore could not be imputed, therefore, such samples would be removed when systematic missingness was observed.

Part 5

The functions of part 5 generate a comprehensive report in .html format. The report includes data quality checks and an exploratory data analysis using valid days of data. First, the numbers and missingness of GGIR output files are summarized for all GGIR parts. Second, duplicate samples were checked and marked, where the duplication might be caused by having multiple visits for some samples but only one visit will be kept in the data analysis such as functional principal component analysis (FPCA). Third, as shown in Figure 1, data quality is presented visually using the number of valid days, non-wear time, and missing data pattern. As an exploratory analysis, the data correlation, and the output of FPCA analysis was plotted in the report.

Part 6 (Optional)

Estimated non-wear periods are loaded from the M$metalong$nonwearscore variable of the R data that was stored in the folder of /meta/basic of the GGIR output, which generate the matrix to clarify when data was imputed for each long epoch time window and the reason for imputation. This function will generate a non-wear matrix at minute level, coded as 0/1 for wear/non-wear time. This function will generate a non-wear matrix at minute level, and it could be skipped if user chose to use the imputation data in the JIVE application as default.

Part 7

In part 7, 88 features were extracted from minute level activity data of three domains of sleep, physical activity, and circadian rhythmicity, which were based on outputs from GGIR v2.4.0 and calculated by R ActFrag and ActCR packages. The standard deviation across days on each subject was also created for each feature. The weekday and weekend specific features were extracted as well since most features in the sleep and physical activity showed significant difference between weekdays and weekends. In brief, sleep domain referred to sleep duration, midpoint, efficiency, etc. Physical activity referred to daily motor activity such as sedentary behavior, light, and moderate-to-vigorous physical activity (MVPA). Circadian rhythms were natural rhythms that regulates the sleep-wake cycle within every 24 hours. For example, the cosinor curve and FPCA analysis were used in modeling of biological rhythms. A comprehensive list of all features could be found in the supplementary table and the detailed definition could be found in the GGIR manual and Di et al.’s publication in 2019.

Input and output of part 0

• Command = call.afterggir(mode=0)
  
• Output folder = ./

part9_swarm.sh | shell script to submit all jobs to the cluster

Input and output of part 1

• Main input files: csv files under /meta/csv folders of GGIR output
• Command = call.afterggir(mode=1)
• Output folder = ./data

f0 and f1 are the file index to start and finish with
Xs is the epoch size to which acceleration was averaged (seconds) in GGIR output

Input and output of part 2

• Main input files
  + ./data/All_studyname_ENMO.data.csv
  + GGIR results: part2, part4 and part5 (please specify part5FN in the main function)
  + GGIR raw data when bindir was specified

• Command = call.afterggir(mode=2)
  

• Output folder = ./summary

Input and output of part 3

• Main input file: ./data/All_studyname_ENMO.data.csv
  
• Command = call.afterggir(mode=3)
• Output folder = ./data

*Xs is the epoch size to which acceleration was averaged (seconds) in GGIR output

Input and output of part 4

• Main input file: ./data/flag_All_ studyname _ENMO.data.5s.csv
• Command = call.afterggir(mode=4)
• Output folder = ./data

Description of flag variables in the output data

Input and output of part 5

• Main input files
  • ./summary/studyname_ggir_output_summary.xlsx
  • ./summary/part24daysummary.info.csv
  • ./data/plot.nonwearVSnvalidhours.csv
  • ./data/impu.flag_All_studyname_ENMO.data.flag.epochOuts.csv
• Command: run part5_studyname_postGGIR.report.Rmd
• Output folder = ./

Input and output of part 6

• Main input file: nonwearscore_studyname_01_xx_900s.csv
• Command = run part6_studyname_postGGIR.nonwear.report.Rmd
• Output folder = ./

f0 and f1 are the file index to start and finish with
Xs is the epoch size to which acceleration was averaged (seconds) in GGIR output

Input and output of part 7a

• Main input
  • ./data/impu.flag_All_studyname_ENMO.data.flag.epochOuts.csv
  • GGIR: /results/QC/part4_nightsummary_sleep_full.csv
• Command = run part7a_studyname_postGGIR_JIVE_1_somefeatures.Rmd
• Output folder = ./

Input and output of part 7b

• Main inputs
  • GGIR: part2_summary.csv
  • GGIR: part2_daysummary.csv
  • GGIR: part4_nightsummary_sleep_cleaned.csv
  • GGIR: /results/QC/part4_nightsummary_sleep_full.csv
  • GGIR: part5_daysummary_part5FN.csv
    
  • part7_studyname_all_features_dictionary.xlsx
• Command = run part7b_studyname_postGGIR_JIVE_2_allfeatures.Rmd
• Output folder = ./

Input and output of part 7c

• Main inputs
  • part7b_studyname_all_features_3_subject.csv
  • part7b_studyname_all_features_4_subjectSD.csv
• Command = run part7c_studyname_postGGIR_JIVE_3_runJIVE.Rmd
• Output folder = ./

Input and output of part 7d

• Main input: ./data/impu.flag_All_studyname_ENMO.data.flag.epochOuts.csv
• Command = run part7d_studyname_postGGIR_JIVE_5_somefeatures_weekday.Rmd
• Output folder = ./

Description of features of domains of physical activity, sleep and circadian rhythmicity

Sleep Domain

Physical Activity Domain

Circadian Rhythmicity Domain