Repository navigation
Job trends in software development by term frequency #226
Description
Overview
Analyze six-month shifts in software development job postings by comparing March 2025 (now a year ago) to the most recent 30 days, quantifying what skills/technologies are rising, falling, newly in demand, or disappearing. Outputs include frequency deltas, trend lines, and a concise narrative report that’s reproducible and portfolio-ready.
The assignee for this issue will have to wear three hats:
- A Data Scientist - Responsible for posing/framing the question(s) and adapting the inquiry according to evidence discovered. An initial hypothes(e)s posed in a report from some intial motivations and investigations. Adapt that hypothesis as insights come in. Create the Final Report for peer review.
- A Data Engineer - Cleaning, Prepping data after devising a strategy to scrape old posting via an API. Implement the scraping and shape the data for visualization by the Data Analyst.
- A Data Analyst - Responsible for answering the Data Scientist's hypotheses, providing a report to them using the insights from the shaped data. Reports to generate along the way.
Action Items
The beginning (research + design)
-
* Define cohort & windows: titles containing
software,developer,engineer,frontend,backend,full-stack,mobile,devops,data engineer,ml engineer; past window = 2022-11-01→2026-03-31, current window = last 30 days. -
* Select sources:
- * Federal: USAJOBS HistoricJoa + AnnouncementText (full text, closed postings).
- * Private sector (optional v1.1): archived postings from Greenhouse/Lever via Wayback snapshots.
-
* Draft skills/tech lexicon with preferred mappings (e.g.,
whitelist→allowlist,primary/replicavsmaster/slave; frameworks: React/Vue/Svelte; platforms: Kubernetes, Terraform; languages: Rust/Go/TypeScript; AI terms: LLM, RAG, vector DB). -
* Methods spec: text extraction & normalization, n-gram and phrase detection, log-odds with informative Dirichlet prior for rise/fall, monthly slope estimation, multiple-testing control (BH-FDR).
-
* Ethics & QA: de-duplicate pages, ignore code blocks, document gaps (unarchived pages), and record rate-limit handling.
If researched and ready (implementation steps)
-
1. Ingest
- Pull metadata with
GET /api/HistoricJoafor both windows; join long fields withGET /api/HistoricJoa/AnnouncementText(key:USAJOBSControlNumber). ([developer.usajobs.gov]1) - (Optional) Enumerate private-sector snapshots via Wayback CDX; select nearest capture to target dates. ([Internet Archive]2)
.
- Pull metadata with
-
2. Extract & Normalize * HTML→main text via trafilatura (fallback: Readability/jusText), lowercase, strip boilerplate & code blocks, Unicode normalize. ([trafilatura.readthedocs.io]3)
-
3.. Feature Build
- Tokenize; build unigrams/bigrams; apply skill lexicon + regex for variants (e.g.,
Type-Script|TypeScript); map deprecated→preferred terms.
- Tokenize; build unigrams/bigrams; apply skill lexicon + regex for variants (e.g.,
-
4. Analysis
- Compute term frequencies, ΔTF-IDF, and log-odds rise/fall between windows; estimate monthly slope on multi-month series if expanded.
- Tag new (present now, absent before) and gone (present before, absent now) terms; flag ambiguous terms for manual review.
-
5. Deliverables
- CSVs:
term_stats_past.csv,term_stats_current.csv,trend_report.csv. - Dashboard (Streamlit or Altair/Vega-Lite): top rising/falling skills, new vs gone, per-title breakdown, sample snippets.
- Methods README with caveats, rate-limit notes, and reproducibility (fixed seeds).
- CSVs:
-
6. Testing & Ops
- Add unit tests for parsing & counting; retries/backoff; de-duplication via URL+digest; logs with timings and error contexts.
- Schedule optional monthly refresh via GitHub Actions (artifact publish).
Resources/Instructions
-
USAJOBS APIs (free, historical)
- API overview & endpoints (Search, Historic JOAs, Announcement Text). ([developer.usajobs.gov]4)
-
Internet Archive / Wayback (for expired private-sector postings)
- CDX Server API reference & help pages. ([Internet Archive]2)
- Python clients: waybackpy or edgi/wayback. ([PyPI]5)
-
Extraction
- trafilatura docs & quickstart; evaluation notes. ([trafilatura.readthedocs.io]3)
-
Schema hints (when scraping non-federal sites)
schema.org/JobPostingand Google’s Job Posting structured data guide. ([Schema.org]6)
-
Suggested stack (choose 1 of each pair)
- HTTP:
httpxorrequests - Parsing:
trafilaturaorreadability-lxml - NLP/Features:
scikit-learnortextacy - Stats:
statsmodelsorscipy - Viz/Dashboard:
Altair/Vega-LiteorStreamlit - Archival:
waybackpyoredgi/wayback
- HTTP:
-
Deliverable spec
- Repo with
fetch/,extract/,analyze/,report/modules;config.yamlfor windows & filters; CI workflow to run end-to-end and publish CSVs + HTML report.
- Repo with
-
If this issue requires access to 311 data, please answer:
- Not applicable. This project uses USAJOBS public APIs and public web archives only.
Activity
15 remaining items
journeywoman commentedon Mar 23, 2026
Progress: Successfully queried the USAJOBS Occupational Series API to filter for IT-sector codes. Developed and executed a script to retrieve historical announcement data via the Historic JOA API for the specified timeframe.
Blockers: None currently. Progress is slightly moderated by the learning curve of navigating public API schemas and managing unstructured data.
Availability: 6–8 hours per week.
ETA: 6-9 weeks.
Next Steps: Parse the retrieved data and extract the necessary information.
journeywoman commentedon Mar 28, 2026
Progress: Parsed the retrieved data. Started visualizing initial datasets.
Blockers: None currently.
Availability: 4–6 hours weekly (reduced due to school spring break). I’ll be away next week, so I won't be able to join the Zoom meeting next Monday.
ETA: 6–8 weeks.
Next Steps: Define AI-related keywords for filtering. Group data by month and year to track the percentage of job postings mentioning AI. Conduct a comparative analysis of average salaries for AI-related roles versus non-AI roles. Verify the hypothesis: AI-related roles should exhibit a higher "Starting Grade" (minimumGrade) compared to the baseline for the same PositionSeries.
Preliminary Results:
journeywoman commentedon Apr 13, 2026
Progress: Completed research based on the Historic JOA API data. I am currently investigating methods to pull information from Greenhouse regarding IT roles.
Blockers: None currently.
Availability: 6 hours per week.
ETA: 6–8 weeks.
Next Steps:
Compile a list of major IT/Tech Greenhouse tokens to seed the research.
Extract a comprehensive list of all unique job titles.
Filter and create a DataFrame of relevant roles.
Develop a targeted script to fetch job descriptions for specific "Data" roles.
Once the full_description column is populated, utilize NLP libraries (e.g., NLTK or spaCy) to extract required skills (Python, SQL, Power BI, etc.) and years of experience.
Preliminary Results:
AI-related roles command significantly higher starting salaries than traditional IT positions within identical series. This indicates that federal agencies formally recognize AI expertise as a high-value, senior-level specialization, justifying a substantial increase in initial fiscal investment per hire.
journeywoman commentedon Apr 20, 2026
Progress: Identified major IT/Tech Greenhouse tokens to initialize the research. Extracted a comprehensive list of all unique job titles and filtered them into a DataFrame of relevant roles. Developed and deployed a targeted script to fetch job descriptions for specific "Data" roles. Successfully extracted key variables including skills (Python, SQL, Power BI), years of experience, salary, country, degree requirements, and work flexibility. Implemented a function to normalize all salary data to USD for standardized analysis.
Blockers: None currently.
Availability: 6 hours per week.
ETA: 6–8 weeks.
Next Steps: I will continue scraping the data and extracting information from job descriptions for analysis. Additionally, I am researching labor market trends using data from the U.S. Bureau of Labor Statistics (BLS) and the Office of Personnel Management (OPM). Compare the ratio of junior-to-senior job postings over the last years to identify potential entry-level displacement. Correlate specific AI-complementary skills (e.g., "Agent Orchestration," "AI Governance") with salary bumps.
journeywoman commentedon Apr 27, 2026
Current Progress
Blockers
There is a lack of available archive data from the Wayback Machine for 2025 and 2026. I am currently investigating the cause (e.g., changes in site crawlability or archival delays) and exploring alternative real-time scraping methods or API-based solutions to fill this gap.
Availability: 6 hours per week.
ETA: 6–8 weeks.
Next Steps
Investigate more efficient scraping alternatives to accelerate the collection process.
Research the cause of missing archive data from the Wayback Machine for 2025 and 2026 (e.g., changes in site crawlability or archival delays) and explore alternative real-time scraping methods or API-based solutions to fill these gaps.
Finalize data collection and begin the formal analysis.
chinaexpert1 commentedon May 12, 2026
Great work. I have more ideas for you in today's meeting.
journeywoman commentedon May 12, 2026
Current Progress:
I have audited data availability for 50 major tech companies. To date, I have successfully retrieved and processed data for 16 companies, including detailed extraction of skills, salaries, locations, and education requirements.
Blockers:
There is a total lack of data availability for the 2025 and 2026 calendar years via the Wayback Machine.
Root Cause Analysis:
The absence of recent data is attributed to several critical challenges currently facing the Internet Archive:
Access Restrictions: Since late 2025, many major domains have implemented "hard blocks" against archive.org_bot to prevent unauthorized AI scraping.
Infrastructure Issues: A confirmed technical breakdown in the Archive’s crawling systems (mid-2025) has significantly reduced snapshot frequency.
Security & Legal Strain: Following a massive 2024 security breach and ongoing multi-million dollar litigation, the organization has diverted engineering resources away from crawler maintenance.
Proposed Solutions:
Direct Scraping: Pivot to scraping live job boards directly to capture real-time data. This will involve establishing a long-term monitoring window (6–12 months) to build a proprietary historical dataset.
Alternative Repositories: Evaluate and integrate Common Crawl datasets to attempt to fill historical gaps where the Wayback Machine is insufficient.
Availability: 6 hours per week.
journeywoman commentedon May 12, 2026
It is a Wayback data.
data_2022_to_2024.xls
@chinaexpert1, please share your thoughts or any specific ideas you have. I’ll review them and incorporate them into the next steps. Thank you!
journeywoman commentedon Jun 9, 2026
Progress: Checked a few hypotheses.
Preliminary Results:
IT Support & Operations shows a massive 22.6% requirement for general "AI", but exactly 0.0% for Machine Learning, Deep Learning, TensorFlow, or NLP. This proves these roles are rapidly adopting AI as an operational tool (likely leveraging LLMs to resolve tickets, draft communications, or automate workflows) without requiring the staff to understand the underlying math or architecture.
Data & Analytics remains the epicenter of AI creation. It is the only category with a real footprint in heavy-compute model building, showing Deep Learning at 11.4% and visible, distinct requirements for specialized frameworks like TensorFlow (4.7%) and Keras (1.5%).
Seeing 17.7% of Cybersecurity jobs ask for "AI" but only 3.8% ask for "Machine Learning" suggests a heavy reliance on automated, AI-driven threat detection software (like advanced endpoint protection). Companies want security analysts who can operate and interpret these AI tools, but they are generally buying them off-the-shelf rather than expecting their in-house security teams to code the models from scratch.
With an Overall AI Penetration of 13.0% and an almost perfectly even split between general "AI" (8.1%) and "Machine Learning" (7.9%), Software Development & Engineering is sitting firmly in the integration phase. Traditional developers are increasingly being tasked with hooking up ML APIs and configuring existing models into web and mobile applications, bridging the gap between the data scientists and the end-users.
AI does not affect the labor market uniformly. For highly technical "producer" roles, AI acts as a ceiling that forces professionals to become more specialized and experienced. For operational "consumer" roles, AI acts as a floor, elevating junior workers and allowing them to enter the field with less prior experience.
Data & Analytics already had the highest baseline for advanced degrees (11.3%), but adding AI pushes it even higher to 13.3%.
Software Development also sees a slight bump (+1.0%), showing that companies view "AI Engineering" as a mathematically heavier discipline than standard software development, often requiring graduate-level computer science knowledge.
For traditional roles like management or general cybersecurity, companies often use a Master's degree as a filtering mechanism. But for applied AI in these same fields, the technology is moving too fast for academia to keep up.
General "AI" keyword demand went from a negligible baseline (under 5%) in 2022 to anchoring over 90% of job postings by late 2024. This is a direct corporate reaction to the mainstream launch of LLMs.
Heavy-compute frameworks like TensorFlow and Keras completely flatlined at the bottom of the graph. Traditional deep-tech model architecture is staying isolated in niche research labs, while the broader IT market values deployment and automation.
Availability & Next Steps: Unfortunately, I need to step away from this project because my main project is taking up all of my time right now. To hand things off properly, I'll package up all my notebooks and findings and upload them sometime this week.
chinaexpert1 commentedon Jun 9, 2026
This is great work. A few things:
Your Wayback data is useful, but includes a lot of internatoinal jobs and I think we want to stay US focused for this report, or you will have scope creep on your analysis. Pleaase see if you can clean the data somehow easily. If not, we probably can't use the data set, unless you want to commit to a larger project that includes global figures. Also, categorizing the jobs would require similar code as cleaning it would.
The AI skills analyis you are doing is very good. However, instead of Tensorflow, please look for Pytorch - it started to take over deep learning workflows a couple years ago.
Your notes on "Barriers to Entry" don't match the chart. Please double check because these are important figures and I want you to include a section on it.
The education wall is very good work. Be sure to note that Data Analytics has more people obtaining advanced degrees, but it is not being as highly compensated for them as your earlier work above shows.
"Companion Skills" section good. Include.
AI skills stuff good, see if you can do a chart for the cybersecurity sector too.
journeywoman commentedon Jun 15, 2026
Here are all the notebooks and files for this project.
historicjoa.ipynb
Historicjoa2.ipynb
Parsing_historicjoa.ipynb
Wayback.ipynb
wayback_2022_to_2024.csv
Tyreal676 commentedon Sep 29, 2026
Progress: Set up google's serpapi to scrape job postings related to data (engineering, scientist, analyst)
Blockers: Not too much of a blocker, just working through the description to start getting skills and technologies for each job.
Availability: 5 hours a week
ETA: Unknown, project is probably going to be continuous/ongoing
Pictures (if necessary):