Neighborhood Council Website technologies

Overview

Project: Open Community Survey

Volunteer Opportunity: Create scraper to get information from builtwith.com on technologies used by neighborhood council websites. Organize the data (create categories for the tech), Automate scrape job to run periodically. Additionally, we want to display this information with a dashboard (see Google Data Studio Dashboard linked below under "Project Output" for an example).

Contact: ~~Ryan Swan (data science), Kaylani (open community survey)~~ Bonnie

Action Items

Resources/Instructions

External Tools

Builtwith
- https://builtwith.com
- builtwith API
  - API limitations: Some sites, are resistant to being crawled (WordPress, for instance https://atwatervillage.org/calendar/). So what we need is a list of all the sites that can't be put through the sitemap maker. See notes about WordPress site crawling: https://community.funnelback.com/knowledge-base/implementation/Gather-And-Index/integration/crawl-wordpress-sites
Selenium
- Selenium Scraping Tools - see branch
Docker

Tutorial

Selenium/Docker, made by Sophia

Project input (data)

Target Website List Here - this is one tab on a larger analysis workbook.

Project output

Rajinder's code

code on data-science repo with Rajinder + Willa's code - this will need to be moved to another directory. It has nothing to do with 311. Its a project for Open Community Survey.
Rajinder's personal repo - this seems to be updated more recently than the one on data-science.

Current presentation

OCS: Tech usage insights NCs
Analytics Analysis Workbook
Widgets Analysis Workbook

Related issues from OCS

NC data scraping task for web technologies open-community-survey#25

Past Collaborators:
@akibrhast, @ava li, @Sarah Williams, @wendywilhelm10 @rajindermavi @ShikaZzz @JessicaFB @Poorvi Rao

@RyanSwan

@RyanSwan @salice @JessicaFB @poorvi4 @akibrhast, @ava li, @sarah Williams, @wendywilhelm10 I forgot to mention that @mattyweb has a report tool that he has setup on our AWS and it might be a good place to dump all this data (from the Comparative analysis of features and then the technologies from builtwith api), then we can define the types of reports we want to display and it would allow people to look at a single NC, as well as aggregate stats, etc. Think of it as a real time data visualizers for end users once we have figured out what is worth looking at. At least that's my understanding of how it works. Would be good to ask him to come to the Data Science Community of Practice at some point to discuss it.

1. Progress (Oct 13, 2025):

Successfully scanned all 97 NC sites; regenerated widgets_report.csv and widgets_summary.csv in out/.
Verified BuiltWith Free endpoint works (manual curl) and confirmed why the parser fails on technology with free payloads.

2. Blockers/Notes:

Free API returns group/category counts (e.g., “widgets → Live Chat / Site Search / Translation / Schedule Management”) rather than per-technology rows, so the old CSV builder crashes. No data loss; just need a tolerant writer.

3. Availability:

Thursday, 10/16/2025 at 4pmPST

4. ETA:

TBD

Next Steps:

Add a small helper (builtwith_free_probe.py) that:
- saves bw_groups_categories.csv
- derives bw_widgets_flags.csv
- summarizes to bw_widgets_summary.csv
Attach heuristic and Free-API summaries to the PR for review.

Progress:
Ran the HTML heuristic; updated counts and examples are in the PR comment.
Blockers/Notes:
Time
Availability:
Thursday, 10/23/2025 at 4pmPST
ETA:
TBD
Next Steps:
Add a small helper (builtwith_free_probe.py) that:
saves bw_groups_categories.csv
derives bw_widgets_flags.csv
summarizes to bw_widgets_summary.csv
Attach heuristic and Free-API summaries to the PR for review.

Progress

Implemented cache-first workflow for BuiltWith Free API (≈1.3s throttle): identify stale/missing caches → refetch targeted batch → rebuild CSVs.
Rebuilt outputs from cache (no Selenium). Current out/ totals:
- bw_widgets_summary.csv → has_calendar=1, has_chatbot=2, has_search=0, has_translation=1
- bw_groups_categories.csv → 330 rows, 9 unique domains parsed

Blockers

Free API payloads are sparse/variable (some domains return no groups), requiring selective refetch.
Rate limit (~1 req/sec) slows full coverage; certain domains may remain unpopulated on the free tier.

Availability

Thursday 11/13/2025: 4–5 pm PT

ETA / Next Steps

Targeted refetch of 15–25 additional domains, rebuild, and refreshed counts by Tue Nov 11, 2025.
If coverage plateaus on the free tier, recommend switching to an org BuiltWith API key (stored as a repo secret) to improve completeness.

@Adaakal

Please provide update @Adaakal

Instructions

Progress: "What is the current status of your project? What have you completed and what is left to do?"
Blockers: "Difficulties or errors encountered."
Availability: "How much time will you have this week to work on this issue?"
ETA: "When do you expect this issue to be completed?"
Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."

You can use this template

1. Progress: 
2. Blockers: 
3. Availability:
4. ETA:
5. Pictures (if necessary):

@Adaakal

Please provide update @Adaakal if you don't respond by next week we will have to give your issue to someone else.

Instructions

Progress: "What is the current status of your project? What have you completed and what is left to do?"
Blockers: "Difficulties or errors encountered."
Availability: "How much time will you have this week to work on this issue?"
ETA: "When do you expect this issue to be completed?"
Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."

You can use this template

1. Progress: 
2. Blockers: 
3. Availability:
4. ETA:
5. Pictures (if necessary):

Progress:

Implemented cache-first workflow for BuiltWith Free API (≈1.3s throttle): identify stale/missing caches → refetch targeted batch → rebuild CSVs.
Rebuilt outputs from cache (no Selenium). Current out/ totals:
bw_widgets_summary.csv → has_calendar=1, has_chatbot=2, has_search=0, has_translation=1
bw_groups_categories.csv → 330 rows, 9 unique domains parsed

Blockers: Unable to continue with the project.
Availability: n/a
ETA: n/a
Pictures (if necessary):

@chinaexpert1

I did some initial investigation into the current web scraping on builtwith.

BuiltWith's current Terms of Use prohibit automated scraping outside of their authorized APIs. Based on earlier comments, it also looks like the project previously explored the limited free API.
I found the official Neighborhood Council directory at neighborhoodempowerment.lacity.gov/councils/. The directory is server-rendered HTML and its /councils/ path is not disallowed by the site's robots.txt.
I tested parsing the directory table directly with BeautifulSoup and was able to successfully extract 99 of the current Neighborhood Council names and website URLs without Selenium.

This looks like a possible path forward: use the official City directory as the source for the current council list, then investigate collecting the needed technology information directly from the council websites rather than BuiltWith.

I have a small proof-of-concept parser working locally and can continue investigating this approach.
@chinaexpert1

@chinaexpert1

@chinaexpert1 I wasn’t able to work on this project this weekend. I had some unexpected new-job onboarding and paperwork come up that took most of my attention. I should be able to get back to the project this week.

@chinaexpert1

Progress:

Switched to the official City Neighborhood Council directory as the source for the current council list.
Implemented a fetch/parser workflow using requests and BeautifulSoup; no Selenium required.
Current directory extraction returns 99 Neighborhood Councils and saves the results to data/councils.csv.
Added logging for fetch timestamp, HTTP status, council count, source URL, and content hash to help detect future directory changes.
Confirmed City directory links lead to individual City-hosted council profile pages.
Tested Arleta, Arroyo Seco, and Arts District Little Tokyo profiles and confirmed they contain links to the councils’ external websites.
Began investigating a reliable way to extract the official council website from each City profile before inspecting those sites for technology information.
Observed a 503 Service Unavailable during repeated requests, so collection will need request pacing/retries and should avoid unnecessary repeat requests.

Blockers: None currently.
Availability: n/a
ETA: none for now
@chinaexpert1

@ExperimentsInHonesty

Progress:

Overview

I've been revisiting the Neighborhood Council website technology assessment to determine whether the original objectives are still relevant and what direction the project should take.

The original goal was to identify technologies and useful website features across LA's 99 Neighborhood Councils, helping councils compare tools, share resources, and potentially adopt features that better serve their communities.

Initial Discovery

See previous progress from the last 3 comments above.
Located the new official City of LA Neighborhood Council directory and confirmed it currently lists 99 councils.
Developed and tested an initial Python workflow using Requests and BeautifulSoup to collect council profile information and discover new external website links.
Added logging to track HTTP responses, timestamps, and changes to the directory.
Reviewed the original comparative analysis spreadsheet, which covers calendars, translation, search, newsletters, CMS platforms, and other website technologies.
Identified an opportunity to refresh the historical website inventory and reassess technologies and features currently in use.

This work is exploratory. No existing project data or production scripts have been replaced.

Blockers

If there is still a need for this assessment:

Validate current Neighborhood Council website URLs against the official City directory.
Update the historical comparative spreadsheet while preserving previous findings.
Develop a reusable process for detecting and categorizing website technologies and features.
Identify useful implementations that other Neighborhood Councils might consider adopting.
Document the collection process and findings for future updates.

Related Issues and Resources

Request for Direction

@ExperimentsInHonesty @chinaexpert1

Before moving further into development, I'd like to confirm whether this assessment is still needed and whether the proposed direction aligns with current Open Community Survey priorities.

Should we continue with the original objectives, revise the scope?

Availability

Monday evenings 7PM

ETA

None for now

Navigation Menu

Repository navigation

Neighborhood Council Website technologies #44

Overview

Action Items

Resources/Instructions

External Tools

Tutorial

Project input (data)

Project output

Rajinder's code

Current presentation

Related issues from OCS

112 remaining items

Progress

Blockers

Availability

ETA / Next Steps

Overview

Initial Discovery

Blockers

Related Issues and Resources

Request for Direction

Availability

ETA

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Neighborhood Council Website technologies #44

Description

Overview

Action Items

Resources/Instructions

External Tools

Tutorial

Project input (data)

Project output

Rajinder's code

Current presentation

Related issues from OCS

Activity

ryanmswan commented on Jul 2, 2021

ExperimentsInHonesty commented on Jul 5, 2021

ExperimentsInHonesty commented on Aug 9, 2021

ebele-oputa commented on Aug 16, 2021

chelseyklein commented on Sep 10, 2021

rajindermavi commented on Sep 11, 2021

112 remaining items

Adaakal commented on Oct 13, 2025

Adaakal commented on Oct 21, 2025

Adaakal commented on Nov 11, 2025

Progress

Blockers

Availability

ETA / Next Steps

chinaexpert1 commented on Jan 20, 2026

chinaexpert1 commented on Jan 27, 2026

Adaakal commented on Jan 27, 2026

parcheesime commented on Aug 26, 2026

parcheesime commented on Sep 22, 2026

parcheesime commented on Sep 28, 2026

parcheesime commented on Oct 15, 2026

Overview

Initial Discovery

Blockers

Related Issues and Resources

Request for Direction

Availability

ETA

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions