Policy Memo #154
As part of the American Reinvestment and Recovery Act of 2009, Congress and the President established a requirement for recipients of Recovery Act funds to report quarterly on a range of activities, including project status, location, and number of jobs created or saved. This (and other) information was to be tracked on a new Web site, Recovery.gov.
On October 15, 2009, Recovery.gov began posting quarterly reports from fund recipients. The first wave consisted of over 5,000 reports by federal contractors who received Recovery Act project funding. Subsequent waves will include over 100,000 reports from recipients of grants and loans. The reports contain a narrative description of the job impact and an estimated number of jobs created or saved.
These reports are a substantial step toward federal accountability and transparency; however, they will not represent a full accounting of the number of jobs created (see Irons 2009). Further, the estimates of the numbers of job created or retained by individual recipients are deeply flawed in many cases.
This Policy Memo outlines some of the data problems concerning the reporting of jobs created or retained. In particular, it highlights several issues, including 1) inconsistent methodologies across recipients; 2) implausible estimates of job creation relative to the size of the grant and the amount of Recovery Act funds received; and 3) internally inconsistent estimates within individual reports, including inconsistencies between the job narrative and the estimated number of jobs gained. Other problems with the data—including incorrect award dates, missing reports, omitted data, Web site interface, etc.—are beyond the scope of this report.
This Policy Memorandum proposes several options to improve the quality and integrity of the data.
Data problems
With the release of the first wave of recipient reports, it is apparent that there are a number of issues with the jobs data as reported by federal contractors. While additional reports on grants and loans are not yet available, there are likely to be similar problems in those reports.
Most of the apparent data issues stem from a failure of contractors to follow guidance (See Appendix A for full guidance language). According to the instructions (Data Model 3.0, derived from Office of Management and Budget (OMB) guidance M-09-21), the estimate of the number of jobs created or saved should be calculated as follows (emphasis added):
At a minimum, this estimate shall include any new positions created and any existing filled positions that were retained to support or carry out Recovery Act projects, activities, or federally awarded contracts managed directly by the recipient or federal contractor. For grants and loans, the number shall include the number of jobs created and retained by sub recipients and vendor. The number shall be expressed as ‘‘full-time equivalent’’ (FTE), calculated cumulatively as all hours worked divided by the total number of hours in a full-time schedule…
However, it should be noted that this failure is, at least in part, due to unclear or insufficient guidance from implementing agencies. For example, the spreadsheet templates (e.g., FederalReportingTemplate-Contract.xls) do not include the full OMB language noted above and only state:
Number of Jobs: Estimate the number of new jobs created and jobs retained in the U.S. and outlying areas. Refer to the Data Model for guidance on how to calculate this number.
Thus, if a contractor only uses the spreadsheet template and does not refer to other guidance (such as OMB M-09-21), they might use their own rule-of-thumb in entering this number. In particular, it is obvious from the first round of reports that many contractors incorrectly assumed that, if work was done with existing workers, then the estimate of the number of jobs would be zero.
While there may be legitimate reasons for some of the data inconsistencies, there appear to be far too many instances where problems arise. Problems include:
• Inconsistent methodologies across recipients. Some recipients report jobs saved and created, others report just new jobs. Some report on sub-contractors’ jobs, others do not. Some convert hours to FTEs, others count heads.
• Implausible estimates of job creation relative to the size of the grant and the amount of Recovery Act funds received. Some contractors show millions of dollars received, yet little or no job impact. Others show little or no funding, but many jobs created.
• Internally inconsistent estimates within individual reports. There are many cases where the estimates of jobs created and saved are inconsistent with other data in the reports. For example:
o Many contractor reports indicate that the project has been partially or entirely completed, and/or has received significant funding to date, yet no jobs have been created.
o Several reports indicate that the project has not yet begun, yet a significant number of jobs have been created.
o The narrative often describes the jobs impact in a way that is not reflected in the estimate of the number of jobs created.
• Contractor reports only jobs created, but omit jobs “saved” or retained. It appears that many recipients are interpreting the jobs created field as measuring only the number of additional people hired by the firm to work on the project. Many contractors are stating that existing employees were used to work on the project and are reporting zero jobs created or saved.
• Unclear if sub-recipient (e.g., sub-contractor) jobs estimates are included. For grants and loans, recipient reports should include these estimates, but contractors don’t necessarily need to report on sub-awards. In general, it is impossible to know if the sub-contractors jobs are included or not, unless specifically included in the job creation narrative.
Going forward, it is also likely that recipients may not report cumulative job creation, but only jobs created in the most recent reporting period. The OMB guidance provides some detail on how the cumulative numbers should be reported; however, given the recent experience, recipients may not be sufficiently informed about the proper method for estimating cumulative job creation.
Classification and examples
Below are some of the main areas in which data appear to be problematic. The first two groupings contain cases that appear to have implausible counts given the ARRA activities, based on simple rules of thumb. The final category contains examples where the narrative does not match the estimate of the number of jobs created. Not every item in these categories is necessarily a problem, and there are problems not captured here; however, these are areas that should be further examined by oversight agencies.
Apparent under-reporting
There are many instances in which contractors report zero jobs, even though they say they have received payments. It is uncommon for federal contractors to receive payments before work has begun, so it is possible that these reflect job-reporting errors.
Moreover, there are many cases in which contractors report zero jobs even though the project status is listed as under way or even completed. Examples of several of these problems are below:
| 1. | Zero jobs reported, but project listed as completed: | non_versioned_guid: | 77591 | ||
| 336 reports | project_description: | As of the end of the calendar quarter (9/30), the project was approximately 100% complete. Only a final inspection remained. A total of 95% complete was invoiced during the calendar quarter. | |||
| job_creation: | No New Jobs were created as a result of this contract. | ||||
| number_of_jobs: | 0 | ||||
| total_fed_arra_received: | $702,287.49 |
| 2. | Zero jobs reported, but recipient received money (>$0): | non_versioned_guid: | 89150 | |||||||||||||||||||||||
| project_description: | Expanded parking lot at child care center | |||||||||||||||||||||||||
| 779 reports | job_creation: | n/a | ||||||||||||||||||||||||
| 3. | Zero jobs reported, but recipient received money >$50k: | number_of_jobs: | 0 | |||||||||||||||||||||||
| total_fed_arra_received: | $79,859.34 | |||||||||||||||||||||||||
| 334 reports | ||||||||||||||||||||||||||
|
| 4. | Zero jobs reported, but recipient received money >$100k | non_versioned_guid: | 78727 | ||
| prime_recipient_name: | Chrysler Group, LLC | ||||
| 224 reports | project_description: | Manufacture and Deliver Vehicles Ordered Under Stimulus Plan | |||
| job_creation: | No jobs were created. Existing employees were utilized to fulfill award orders. | ||||
| number_of_jobs: | 0 | ||||
| total_fed_arra_received: | $52,892,867.00 | ||||
| non_versioned_guid: | 61786 | ||||
| project_description: | Payment for reconstruction of road and assistance to Senior Center for operations. |
||||
| job_creation: | No new jobs were created with this funding. | ||||
| Number_of_jobs: | 0 | ||||
| Total_fed_arra_received: | $102,086.84 | ||||
| non_versioned_guid: | 89086 | ||||
| project_description: | |||||
| 1004AA Remove Curb & Gutter, 1005AA Remove Valley Gutter or SIdewalk, 1007AA Ditch Grading, 1008AA Excavation/Disposal, 1010AA Compacted Fill, 1012AA Curb & Gutter, 1014AA Sidewalk, 1017AA Compacted Sub-base, 1017AB Stablizied Aggregate Base, 1018AA Tack Coat, 1022AA AC Pavement, 1025AA Raise Utility Covers, 1028AB Traffic Striping,1028AE Striping & Reflector Media, 1031AA Landscaping/Hydroseed, 1032AA Soils Test, 1034AA Moisture Density Relationship, 1039AC Asphalt Test, 1039AD Asphalt Test, 1064AA Clear & Gurb, 1078AA Traffic Control | |||||
| job_creation: | Operator Engineers; Carpenters; Laborers; Project Managers; Quality Control Personell | ||||
| number_of_jobs: | 0 | ||||
| total_fed_arra_received: | $495,119.85 | ||||
| 5. | Large contract/grant award (>$1 million received), few jobs reported (<2): | non_versioned_guid: | 33916 | ||
| job_creation: | construct remainder of earthen closure | ||||
| 50 reports | number_of_jobs: | Quality Control Manager | |||
| number_of_jobs: | 1 | ||||
| total_fed_arra_received: | $1,568,088.04 | ||||
| non_versioned_guid: | 81571 | ||||
| project_description: | Boring and electrical lines in progress | ||||
| job_creation: | Construction manager/office staff | ||||
| number_of_jobs: | 0.54 | ||||
| total_fed_arra_received: | $4,300,768 | ||||
| non_versioned_guid: | 81210 | ||||
| > | project_description: | Mobilized lab at Augusta, Georgia location. Prepared tutorials, began hiring and training curation and artifact and records processing staff, purchased all equipment and furnishings. | |||
| job_creation: | Created work for senior program manager, senior project managers, laboratory director, laboratory supervisors. | ||||
| number_of_jobs: | 1.73 | ||||
| total_fed_arra_received: | $1,247,080 | ||||
| 6. | Sub-contractors have a large share of total contract (>75%, and $100k), few jobs reported (<1)
35 reports |
non_versioned_guid: | 46217 | ||
| project_description: | Project management for the recomissioning process for the Skaggs Research Center. This inculdes working in unison with the firm hired to perform the recomissioning process, which includes communication with them and ensuring dates and deliverables are being met. | ||||
| number_of_jobs: | 0.01 | ||||
| award_amount: | $114,126.54 | ||||
| total_fed_arra_received: | $67,314.32 | ||||
| non_versioned_guid: | 61568 | ||||
| project_description: | Roofing Repairs, Carpet Replacement, Repaving of Parking Lots, Painting of Facilities, Erecting Facilities and Restroom Renovation | ||||
| job_creation: | No jobs were created under the Prime Contract as a Result of these funds. | ||||
| number_of_jobs: | 0 | ||||
| total_fed_arra_received: | $2,914,255.03 | ||||
| non_versioned_guid: | (73870) | ||||
| project_description: | Prepped & pressure washed buildings 1867,1869,1835,1830,1831. Repaired, Primed and Painted walls. Currently prepping and pressure washing Building 1840. A snorkel left was delivered on 6 Oct 09 to assist with building 1840. | ||||
| job_creation: | No new jobs created. | ||||
| number_of_jobs: | 0 | ||||
| total_fed_arra_received: | $91,730.69 | ||||
Apparent over-reporting
In many cases, firms may have reported “too many” jobs relative to their funding. Some recipients may report on new temporary or part-time hires (rather than converting hours worked to FTEs). Note that these cases may not necessarily indicate problems; it may be the case that firms have hired in anticipation of receiving money.
| 1. | Small contract/grant (<$50k), many jobs reported (>2) |
non_versioned_guid: | 78845 | |||||||||||||||||
| 130 reports | job_creation: | design, engineering, testing, and construction. | ||||||||||||||||||
| number_of_jobs: | 10 | |||||||||||||||||||
| award_amount: | $46,072.32 | |||||||||||||||||||
|
| 2. | Number of jobs > 0, but
recipient received no money: |
non_versioned_guid: | 25917 | |||||||||||||||||
| job_creation: | Jobs on this project are for the construction of rail track and signal work completed by the Union Pacific Railroad. | |||||||||||||||||||
| 925 reports | ||||||||||||||||||||
| number_of_jobs: | 125 | |||||||||||||||||||
| total_fed_arra_received: | $0 | |||||||||||||||||||
|
| 3. | Number of jobs > 0, but
project has not been started: |
non_versioned_guid: | 47003 | |||||||||||||||||
| job_creation: | jobs created for this project are for three laborers to apply pesticides in the national forest over the course of three weeks time. | |||||||||||||||||||
| 399 reports | ||||||||||||||||||||
| number_of_jobs: | 3 | |||||||||||||||||||
| project_status: | 1
d> |
|||||||||||||||||||
|
Description does not match count
Aside from the potential problems noted above, there appear to be many cases in which the narrative description does not match the jobs estimates. This appears to be particularly true in cases in which jobs were saved rather than created.
| non_versioned_guid: | 48273 | |||||||||||||||||||
| job_creation: | Jobs are performed by Berger employees and subcontractor employees. No additional jobs created. | |||||||||||||||||||
| number_of_jobs: | 0 | |||||||||||||||||||
| total_fed_arra_received: | $81,442.80 | |||||||||||||||||||
|
| non_versioned_guid: | 61509 | ||||||||||||||||||||||
| job_creation: | No jobs created. Retained 0.19 FTEs during the quarter for biologists, scientists and technicians. Employment impact was neglible. | ||||||||||||||||||||||
| number_of_jobs: | 4.19 | ||||||||||||||||||||||
| award_amount: | $16,073.25 | ||||||||||||||||||||||
|
Recommendations
It is clear from the recipient reports that either the jobs-reporting instructions are unclear, that recipients are ignoring the guidance, or both. The proposals below would 1) improve the guidance and communications with recipients; 2) help prevent problematic data from being entered into the system; 3) flag potential problems for additional scrutiny; and 4) augment reporting to include data that would be more likely to result in consistent estimates.
• Improve the reporting process to clarify and re-emphasize jobs-reporting guidance. In particular, guidance and instructions need to be clarified and emphasize that all hours of work on ARRA projects, whether performed by existing workers or new hires, should go into the job calculations.
• At a minimum the spreadsheet templates, XML Schema, online forms, etc., should all include the guidance language that states the method for calculating the number of jobs created or retained. In particular, the instructions should include: “The number shall be expressed as ‘‘full-time equivalent’’ (FTE), calculated cumulatively as all hours worked divided by the total number of hours in a full-time schedule.”
• OMB and FederalReporting.gov guidance should be amended to include examples to cover various scenarios, with a particular emphasis on how to report on the number of jobs saved or “retained.”
• Stand-alone materials should be developed to illustrate the methods that should be employed. These materials should again include examples to clarify that the proper method is to first estimate the number of hours worked on the project.
• The recipients should be given the option to enter the total number of hours worked on the project and an estimate of the total number of hours in a FTE over that period. The FTE hours should also be checked to ensure that the number is plausible (e.g., reported in hours per day, or hours per week, or hours per quarter).
• FederalReporting.gov should also prepare a supplemental worksheet (either online or downloadable) that walks recipients through the appropriate process for calculating the number of jobs (see Appendix B for a short example).
• Screen recipient data for obvious errors. Recipient reports should be automatically screened for obvious errors. The Recovery Board already conducts a first review to ensure that recipient reports do not include, for example, millions of jobs created. This process should continue to be refined and, if not done already, systemized.
• Flag recipient reports which are very likely to contain errors or misinterpretations of jobs instructions. The counts of reports below only include the first wave of recipient reports from contractors (and thus do not include loan and grant recipients).
Red Flag: (High probability of inaccurate jobs data.)
• Zero jobs reported, but project listed as Completed: 336 reports
• Zero jobs reported, but recipient received money >$50,000: 334 reports
• Large contract/grant award (>$1 million received), few jobs reported (<2): 50 reports
• Sub-contractors have a large share of total contract (>75%, and $100k), few jobs reported (<1): 35 reports
• Jobs narrative contains phrases like: “prevented layoffs,” “existing staff,” “existing positions,” and “current employees,” while the number of jobs reported is small relative to grant (<2 or fewer than 1 per $200,000 in received funds).
Yellow Flag: (Substantial probability of inaccurate jobs data).
• Zero jobs reported, but recipient received money >$0, but <$50,000: 445 reports
• Small contract/grant (<$50k), many jobs reported (>2): 130 reports
• Number of jobs > 0, but recipient received no money: 925 reports
• Number of jobs > 0, but project has not been started: 399 reports
• Improve report validation. When submitting a report that would trigger a flag as noted above, the recipient should be immediately notified of the potential flag and given the option to revise their reports before submitting a final report. The notice should also include the text of the jobs-reporting guidance:
• For future quarterly
reports, the entries should be checked to ensure that the jobs estimate is at least as large as prior reports. This will provide an initial check on whether the recipient is reporting on cumulative job creation and retention.
• Modify Recovery.gov to indicate recipient reports that are flagged. Recovery.gov should augment all reports, downloads, etc., to include Recovery Board flags as noted above. These data should also be provided to agencies for further follow-up if necessary. Separate files of only the flagged reports should be provided to assist public scrutiny of the reports.
• Add data fields on jobs reporting. Additional fields would help reduce ambiguities and clarify reporting methodology:
• Does job estimate include subcontractors’ jobs (yes/no)? If yes: how many (#)?
• Does job estimate include saved jobs (yes/no)? If no, immediately red flag the report and reiterate guidance.
• Estimated total cumulative number of hours worked on Recovery Act–funded project(s).
Conclusion
The first wave of recipient reports posted on Recovery.gov provides a wealth of data for observers to sift through on a range of Recovery Act funding. However, it is now apparent that the jobs data are of only limited value because of a range of issues from inconsistent methodologies for estimating jobs to implausible reports of jobs created or retained.
Implementing the recommendations above—from clarifying guidance to flagging and validating reports—would go a long way toward improving the quality of the data, both in aggregate and for individual recipient reports.
—The author would like to thank Gary Bass, Craig Jennings, Greg Leroy, Phil Mattera, and Ethan Pollack for helpful comments and discussions.
Appendix A: Data model 3.0
Recipient Reporting Data Model 3.0 (accessed 10/25/09)
Number of Jobs
“Jobs created and retained. An estimate of the number of jobs created and jobs retained in the United States and outlying areas.
At a minimum, this estimate shall include any new positions created and any existing filled positions that were retained to support or carry out Recovery Act projects, activities, or federally awarded contracts managed directly by the recipient or federal contractor. For grants and loans, the number shall include the number of jobs created and retained by sub recipients and vendor. The number shall be expressed as ‘‘full-time equivalent’’ (FTE), calculated cumulatively as all hours worked divided by the total number of hours in a full-time schedule, as defined by the recipient or federal contractor.
For instance, two full-time employees and one part-time employee working half days would be reported as 2.5 FTE in each calendar quarter. A job cannot be reported as both created and retained. As used in this instruction, United States means the 50 States and the District of Columbia, and outlying areas means…”
Description of Jobs Created
“A narrative description of the employment impact of the Recovery Act funded work. This narrative is cumulative for each calendar quarter and at a minimum, will address the impact on the recipient’s or federal contractor’s workforce (for grants and loans, recipients shall also include the impact on the workforces of sub recipients and vendors).
At a minimum, provide a brief description of the types of jobs created and jobs retained in the United States and outlying areas. ‘‘Jobs or positions created’’ means those new positions created and filled, or previously existing unfilled positions that are filled, as a result of Recovery Act funding. ‘‘Jobs or positions retained’’ means those previously existing filled positions that are retained as a result of Recovery Act funding. This description may rely on job titles, broader labor categories, or the recipient’s existing practice for describing jobs as long as the terms used are widely understood and describe the general nature of the work.
ALTERNATE METHOD FOR GRANT AND LOAN RECIPIENTS: In those circumstances where the recipient employs an approved statistical methodology to generate estimates of job impact, thereby collecting data from a smaller subset of sub-recipients and vendors in order to extrapolate an estimate of job impact to all applicable sub recipients and vendors, the recipient must provide a description of the methodology used.”
Appendix B:
Example worksheet for calculating jobs created or retained.
| Method 1. | |
| Enter the total number of hours of work performed on Recovery Act funded projects | 120 |
| Enter the number of hours in a Full-time job (per week) | 40 |
| Enter the number of weeks in the reporting period | 13 |
| Number of Jobs Created and Retained | 0.23 |
| Method 2. (Colleges and Universities) | |
| Enter the number of FTE employees engaged in Recovery Act funded project(s) | 10 |
| Enter the average percentage of time in the reporting period spent on Recovery Act funded project(s) | 25% |
| Number of Jobs Created and Retained | 2.5 |
References
Irons, John S. (2009) “Recovery.gov’s Jobs Data: A (Very) Partial Monty.” Economic Policy Institute, October 14, at http://www.epi.org/analysis_and_opinion/entry/recovery.govs_jobs_data_a_very_partial_monty/.
